Angelica M. Tinga
Tycho T. de Back- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, Netherlands
Although many scholars deem non-invasive measures of neurophysiology to have promise in assessing learning, these measures are currently not widely applied, neither in educational settings nor in training. How can non-invasive neurophysiology provide insight into learning and how should research on this topic move forward to ensure valid applications? The current article addresses these questions by discussing the mechanisms underlying neurophysiological changes during learning followed by a SWOT (strengths, weaknesses, opportunities, and threats) analysis of non-invasive neurophysiology in learning and training. This type of analysis can provide a structured examination of factors relevant to the current state and future of a field. The findings of the SWOT analysis indicate that the field of neurophysiology in learning and training is developing rapidly. By leveraging the opportunities of neurophysiology in learning and training (while bearing in mind weaknesses, threats, and strengths) the field can move forward in promising directions. Suggestions for opportunities for future work are provided to ensure valid and effective application of non-invasive neurophysiology in a wide range of learning and training settings.
Introduction
While behavioral methods, such as pre- and post-test assessments, are most commonly used to assess learning, non-invasive neurophysiological methods provide promising complementary options (Leff et al., 2011; Lai et al., 2013; Krigolson et al., 2015; Tinga et al., 2019a). Non-invasive neurophysiology includes measures that provide insight into the nervous system through relatively non-invasive sensors placed on the body or in the environment. Examples of these measures are heart rate, electrodermal activity (EDA) and electroencephalography (EEG).
Although many scholars deem these measures to be promising in assessing learning, they are currently not widely applied in educational settings and in training. This article aims at providing insights that help the field move toward valid and effective application of neurophysiology in a wide range of learning and training settings. In the current article learning is considered as processing of information from experiences in order to update system properties (Barron et al., 2015). Through experience the ability to execute tasks or operations or processing of information improves and becomes faster, less effortful and more automatic (Schneider and Chein, 2003; Borghini et al., 2016). A large body of cognitive and neurocognitive studies demonstrates that learning manifests itself in different ways, such as the acquisition of new information, the development of perception, reasoning, psychomotor skills, and problem solving skills (Lai et al., 2013). When referring to learning in the current article we specifically refer to psychomotor and cognitive learning, but generalize to other types of learning. Affective learning, such as fear conditioning and fear extinction, however, is not included as this type of learning falls outside of the scope of the current article, as studies on this topic primarily reflect the affective response to stimuli. To gain insight into how we can move in the direction of valid and effective application of neurophysiology in a wide range of learning and training settings, it is first of all important to know the underlying mechanisms of how non-invasive measures of neurophysiology are able to inform on learning. To this aim we will first discuss the most frequently applied non-invasive measures and into what aspects of the nervous system these measures provide insight. Subsequently, we will discuss how activity in the nervous system, in turn, can provide insight into learning processes.
The nervous system consists of the autonomic and central nervous system. The autonomic nervous system has two branches that operate at the same time, but in an antagonistic way: the sympathetic nervous system and the parasympathetic nervous system, in which the sympathetic branch activates a neurophysiological response, while the parasympathetic branch inhibits one (Berntson et al., 1991), with parasympathetic nerves exerting their effects faster than sympathetic ones (Nunan et al., 2010). The nervous system regulates a wide variety of functions related to processes such as homeostasis, digestion and attention (Berntson and Cacioppo, 2000; Mauss and Robinson, 2009).
Non-invasive methods providing insight into the autonomic nervous system include measures of peripheral physiology such as heart rate (variability), respiration, EDA and electromyography (EMG). Heart rate and heart rate variability (i.e., the variation in the intervals between consecutive heartbeats) can be measured either via electrocardiography (ECG) or photoplethysmography (PPG). While ECG is more conventional, PPG has the benefit of being relatively less invasive. With ECG multiple sensors are placed on the body to measure the heart’s electrical activity, while with PPG a single optical sensor is placed on the finger or earlobe in order to measure peripheral changes in blood flow which are affected by activity of the heart. Although ECG is considered to be more precise than PPG, especially in measuring heart rate variability, the two measures are highly correlated under ideal conditions (Lu et al., 2009). Respiration can be measured through a respiration belt around the upper body measuring the body’s expansion and contraction. EDA refers to variations in the electrical properties of the skin in response to sweat secretion. A non-invasive measure of the change in conductance of the skin can be acquired by applying a low constant voltage across two electrodes placed on the skin (Fowles et al., 1981). EMG provides insight into muscle activity as this method measures electrical activity that is generated by muscle fibers. While the heart and the lungs are connected to the sympathetic as well as to the parasympathetic nerves, the skin and its sweat glands and skeletal muscles are solely innervated by the sympathetic nerves (Roatta and Passatore, 2008; Setz et al., 2010).
The central nervous system refers to the part of the nervous system consisting of the brain and the spinal cord. Brain activity can be measured non-invasively through measures such as EEG and functional near-infrared spectroscopy (fNIRS). fNIRS is a method in which cortical hemodynamic activity is measured using near-infrared light and is based on the assumption that neural activation and the vascular response are coupled (León-Carrión and León-Domínguez, 2012). EEG records cortical electrical activity non-invasively through electrodes on the scalp (Antonenko et al., 2010). EEG is applied more often than fNIRS, as fNIRS is still a relatively new measurement technique (Kopton and Kenning, 2014). Compared to EEG, fNIRS has a better spatial resolution but a reduced temporal resolution (Crosson et al., 2010; Zama and Shimada, 2015). Other well-established techniques of measuring brain activity include functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). However, these measures tend to be more invasive (Tinga et al., 2019a) and therefore fall outside the scope of the current review.
Eye-related measures are another category of neurophysiological measures. The eye is affected both by the central and autonomic nervous system; the retina is an integral part of the central nervous system, while the muscles that control eye-movements, eyelid elevation and pupil dilation are innervated by both branches of the autonomic nervous system (McDougal and Gamlin, 2015; Eckstein et al., 2017; Larsen and Waters, 2018). Eye-related measures can be collected non-invasively through an eye-tracker either placed on a table in front of or mounted on the head of the trainee. Alternatively, sensors can be placed on the skin close to the eyes to measure the electrical activity that is paired to the eye’s position and movements, a technique called electrooculography (EOG). Eye-related measures include (1) pupil dilation; (2) eye gaze; (3) blink rate; and (4) blink duration (McDougal and Gamlin, 2015; Eckstein et al., 2017). Why would the activity in the nervous system measured through neurophysiology be related to learning processes? Measures of neurophysiology respond in a predictable manner to cognitive demand/mental workload or mental effort (Gevins and Smith, 2003; Antonenko et al., 2010; Setz et al., 2010; Ayaz et al., 2012; Brouwer et al., 2012; Borghini et al., 2014; Hogervorst et al., 2014; Reiner and Gelfeld, 2014; Gable et al., 2015; Charles and Nixon, 2019). As mental effort increases, sympathetic activation increases and parasympathetic inhibition decreases (Berntson et al., 1991), paralleled by changes in the central nervous system such as in oscillations in the EEG signal with alpha oscillations generally increasing and theta oscillations generally decreasing with decreasing demands (Klimesch, 1999; Klimesch et al., 2005; Antonenko et al., 2010; Brouwer et al., 2012) and in activation patterns in the fNIRS signal with activity in the (pre)frontal cortex generally increasing (Ayaz et al., 2012; Hiyamizu et al., 2014).
According to several influential cognitive theoretical frameworks learning can be thought of as a change in how a to-be-learned task is processed and therefore in how cognitively demanding a task is. The dual processing theory constitutes one of these frameworks. From the dual processing perspective, learning can be thought of as a transition from controlled to automatic processing (Schneider and Chein, 2003). Controlled processing is slow, effortful and attentionally demanding and is comparable to a high mental effort. Automatic processing, in contrast, is fast and occurs in the absence of control and attention and is comparable to low mental effort. Evidence for the transition from slow controlled processes to faster automatic processes comes from behavioral findings demonstrating that the beginning of the learning process is characterized by slow and often inaccurate responses (Schneider and Shiffrin, 1977) and that when learning advances the time needed for task execution decreases (Ritter and Schooler, 2001).
Similarly, from the perspective of cognitive load theory learning can be seen as the process of schema acquisition and schema automation (Schnotz and Kurschner, 2007), whereby items that were first processed separately become integrated within one schema and regarded as a single item. In this way, fewer items need to be held in working memory, reducing mental effort. Processing of the acquired schema can in turn become automatic during further learning. This schema automation reduces working memory demands or demands on cognition even more.
Mental effort additionally increases with increased task difficulty (Backs and Seljos, 1994; Hockey, 1997; Brouwer et al., 2014) and task difficulty is thought to modulate learning-related changes in neurophysiology (Wang et al., 2010). In general, execution of a task becomes less difficult during learning, eliciting changes in mental effort and therefore neurophysiology (Fairclough et al., 2005). Moreover, according to motivational intensity theory mental effort is proportional to the level of task demand provided that success is possible and successful performance is worthwhile (Brehm and Self, 1989). This has been demonstrated on several neurophysiological measurements as well (Richter et al., 2008, 2016; Fairclough and Ewing, 2017), implying that success needs to be possible in order for learning to occur and for mental effort and neurophysiology to change in a predictable way during learning.
The topic of learning consists of various aspects of development in cognition. It is important to take into account that different aspects of learning progress differently and mostly do not affect cognitive development in a linear fashion (Schneider and Chein, 2003; Luft and Buitrago, 2005; Murre, 2014; Stoianov et al., 2016). Even though cognitive development does not necessarily progress in a linear fashion, it is clear that important cognitive changes take place during learning, transitioning from more controlled and demanding processing to less demanding automation, which can be detected by neurophysiological measures. Experimental evidence comes from studies demonstrating changes in neurophysiology suggesting a decrease in mental effort over time during learning (for an overview of 113 experiments from 2006 to 2016 and a first model based on the findings from these experiments see Tinga et al., 2019a). Generally, in such experimental studies, a learning task is presented in which improvements over time (e.g., trials or blocks) on behavioral outcomes occur and the effect over time on simultaneously recorded neurophysiological measures is assessed and related to the behavioral learning process. For example, in a recent study including two experiments (Tinga et al., 2020) participants were presented with a visuomotor sequence learning task while behavioral measures and measures of EEG, skin conductance, heart rate (variability) and respiration were collected. Results indicated that, in both of the two experiments, skin conductance level and EEG oscillations (in the alpha and gamma band) changed during general task learning indicating less mental effort over time and were related to behavioral performance. These findings suggest neurophysiology is able to provide robust insights in learning. Moreover, several studies have compared learning and non-learning task conditions and report a clear difference on neurophysiology (Krigolson et al., 2009; Alain et al., 2010; Takeuchi et al., 2011; Moisello et al., 2013; Tan et al., 2014), supporting that there is potential in these measurements as they are able to dissociate learning from non-learning.
Different measures of neurophysiology all provide insight into activity in a part of the nervous system, yet there are important differences in specific challenges associated with each measurement technique. For example, measurements of heart rate variability are sensitive to whether the trainee is sitting, standing or lying down (Acharya et al., 2005). As another example, measures of pupil size are affected by the luminance of the surroundings (Höfle et al., 2008). Even though important differences between the different neurophysiological measures exist, we will consider all measures together in this review. These different measures of neurophysiology are rarely considered together, while it has been pointed out that it is important to discover what specific aspects of the learning process are reflected in specific neurophysiological changes (Tinga et al., 2019a). Moreover, different measures of neurophysiology can potentially be combined to provide better insight into learning than a single measure (cf. Wilson and Russell, 2003, 2007). However, where needed, important differences between different measures of neurophysiology will be discussed.
Even though neurophysiological measures have potential to provide insight into learning, the question remains how research on this topic should move forward to ensure valid applications. The nature of this current overview does not allow to go into the details of methodological issues inherent to the studies reviewed, but will instead assume the importance of these peer-reviewed articles and focus on how to best move the field forward.
A Swot Analysis for the Field of Non-Invasive Neurophysiology in Learning and Training
To gain a better understanding of how the field of non-invasive neurophysiology can best move forward, it is important to comprehensively assess the current state of the field. To this aim, we will identify strengths, weaknesses, opportunities, and threats (SWOT) for the field of non-invasive neurophysiology in learning and training. Although the SWOT framework is most commonly employed in business for analyzing factors that influence a company’s position in the marketplace with a focus on the future, it can also be useful for other domains such as in a scientific field (Rizzo and Kim, 2005). Before we discuss each of the strengths, weaknesses, opportunities and threats, we present a summary of the analysis as a guideline (Table 1).
Table 1. Summary of the SWOT analysis for the field of non-invasive neurophysiology in learning and training, with each of the items discussed in the text.
Strengths in Applying Neurophysiological Measures in Learning and Training
Formative and Objective Assessment Supporting Predicting the Outcome of Learning
Learning can be assessed in a summative or formative way. Summative assessment focuses primarily on assessing the outcome of learning while formative assessment aims to gain insight into learning processes (Stobart, 2008). While summative assessment informs on whether learning is sufficient, formative assessment additionally allows for supporting learning based on the examination of the learning process itself. Learning is commonly evaluated using summative measures of behavioral performance by for example using a post hoc written examination (Weinstein, 2001) focused on the outcome of learning while neglecting the learning process itself. Having insight into the learning process could even support predicting the outcome of this process even before someone has finished learning (Cowley et al., 2013; Vecchio et al., 2019).
Additionally, assessment needs to be objective, valid, and reliable (Brenner et al., 2015). Post hoc behavioral measures might lack objectivity and validity as they might be affected by the process of measurement itself with participants changing their behavior on purpose or without being conscious about it when they are able to detect that specific behaviors are being measured (Page et al., 1966). Non-invasive neurophysiological measures can be an interesting substitution or addition to behavioral outcomes of learning by being a formative assessment tool providing insight into the learning process online in an objective way.
Supporting Learning Through Instruction or Feedback and Tailoring Task to the Individual
Providing feedback during learning can greatly benefit learning (Hattie and Timperley, 2007; Sigrist et al., 2013). As neurophysiology is a formative measure, it can be employed for real-time support of learning. When measures indicate that the trainee is having difficulty with learning, the trainer can provide additional instruction and feedback. Potentially, automated feedback as a response to incoming neurophysiological data and additional task data could even be implemented in the learning task itself. The continuous stream of objective data could also be used for personalization of the learning environment to the needs of the individual in order to ensure that learning is neither too easy nor too difficult. For example, the learning task could be made more challenging when the trainee is learning sufficiently, but when learning is insufficient more challenging material could be presented at a later point in time or the task could be made simpler. Walter et al. (2017) studied this principle using EEG data during arithmetic learning in students. They adapted the difficulty of the material to the learner’s workload and concluded that such an adaptive learning environment is feasible. Additionally, systems are being developed in which eye tracking data is employed in order to track learning and support learning through feedback and adaptation in e-learning environments (Barrios et al., 2004; Song et al., 2015).
Multi-Channel Data Collected at a High Sample Rate Providing Rich Data
Non-invasive neurophysiology allows for measuring through multiple channels within one modality as well as across a range of modalities. Channels can be combined in order to reach conclusions of enhanced reliability as it has been demonstrated by Wilson and Russell (2003, 2007) that mental effort could be classified using a combination of neurophysiological measures (however, see Hogervorst et al., 2014; Tinga et al., 2020 who did not find that combining different neurophysiological measures improved assessment of mental effort and learning specifically). Besides the potential of enhancing reliability, if one channel fails to record sufficient data there are other channels to fall back on. Additionally, data can be collected at a high sample rate during the learning experience. For example, the sample rate typically ranges from 250 to 2000 Hz for EEG (Reis et al., 2014), from 25 to 2000 Hz for eye tracking (Reis et al., 2014), and from 500 to 1000 Hz for heart rate (Kwon et al., 2018). Although a higher sample rate does not always provide extra information (Reis et al., 2014; Kwon et al., 2018), a relatively higher sample rate allows for a precise description of processes over time. Data provided by frequently sampled multi-channel neurophysiological methods is therefore much richer than a single post hoc behavioral measure such as an exam score.
Flexible and Precise Application and Measurement
Another benefit of neurophysiological methods is that there is flexibility in where each sensor will be placed with multiple ways in which one modality can be measured while still recording with high precision. When measuring brain activity through EEG or fNIRS for example, one can decide to place sensors only at specific locations on the scalp or on locations distributed over the scalp. EEG sensors have an outstanding temporal resolution providing insights in the temporal dynamics of neuronal activity below 100 ms. Additionally, EEG sensors allow for source localization with adequate precision, especially from 64 electrodes, with a higher number of electrodes not being necessary (Baillet et al., 2001; Michel et al., 2004; Michel and Murray, 2012). For applications other than source localization, it is not required to include a large number of electrodes for measuring EEG, with studies reporting successful mental effort assessment through a single EEG sensor (Mak et al., 2013; Hogervorst et al., 2014). Regarding measurements such as heart rate and EDA, different configurations of electrode placement are possible (Francis, 2016), providing flexibility across a range of tasks. For example, heart rate, while traditionally measured via sensors on the chest, can alternatively be measured on the ankle/foot (Díaz and Pallas-Areny, 2010; Jarchi and Casson, 2016). As EDA works by measuring the change in conductance of the skin (Fowles et al., 1981), several different placement locations are possible. Cameras collecting eye tracking measures can be integrated into glasses worn by the trainee or can be positioned at a distance from the trainee (Meißner et al., 2019), as long as the cameras can reliably detect the eyes. Alternatively, sensors can be placed on the skin close to the eyes (EOG). Thus, there is a lot of flexibility in the type of sensors and the location of placements of the sensors, with measurements being able to be highly precise when they are suited to the goal of the measurements.
Weaknesses in Applying Neurophysiological Measures in Learning and Training
Neurophysiology Is Reflective of a Wide Variety of Functions Supported by the Nervous System
Besides the important strengths associated with applying non-invasive measures of neurophysiology in learning and training, there are also important weaknesses. One weakness is inherent to the general-purpose nature of the nervous system. Its activity is not reflective of one process, but rather of a wide variety of functions related to homeostasis, digestion, attention, and so forth (Berntson and Cacioppo, 2000; Mauss and Robinson, 2009), which is an important point to keep in mind when interpreting measures reflective of nervous system activity. Although learning affects neurophysiology and these effects have been detected across several studies (Krigolson et al., 2009; Alain et al., 2010; Takeuchi et al., 2011; Moisello et al., 2013; Tan et al., 2014), neurophysiology is additionally sensitive to many learning-irrelevant processes and events. For example, pupil size is determined to a great extent by luminance of the surroundings making it important to control for luminance when interpreting pupil size data (Höfle et al., 2008). Effects of ambient light have also been reported for other neurophysiological measures, such as heart rate, with effects depending on the time of day (Ruger et al., 2006). Additionally, effects of emotion and stress (Mauss and Robinson, 2009) and mental fatigue (Gergelyfi et al., 2015) on measures of neurophysiology have been reported frequently. As learning is a process involving prolonged cognitive activity, mental fatigue can be induced in the trainee. Learning and mental fatigue are thought to both progress over time and therefore disentangling the two is important in order to determine which changes in neurophysiology are truly reflective of learning. Gergelyfi et al. (2015) experimentally demonstrated (as the first to do so according to the authors) that pupil size, skin conductance, heart rate, and EEG were unaffected by mental fatigue while blink rate and heart rate variability were affected by mental fatigue. These findings suggest that different neurophysiological measures are affected by mental fatigue in different ways, although more research is needed in order to be able to deal with effects of mental fatigue sufficiently.
Neurophysiological Changes During Learning Are Sensitive to Individual Differences and Task-Related Aspects
Changes in neurophysiological outcome measures during learning are also affected by individual differences and aspects of the learning task (Tinga et al., 2019a). For example, several studies have demonstrated changes in brain activity as assessed through EEG during learning to be different between young and older adults (Alain and Snyder, 2008; Eppinger et al., 2008; Pietschmann et al., 2008, 2011; Eppinger and Kray, 2011). Effects of age on learning-related changes in neurophysiology have not yet been studied for other outcome measures than measures of brain activity. It therefore is unclear whether age also affects measures of peripheral physiology. Neurophysiological changes during learning are also reported to be affected by the sensory system in which learning takes place, the number of learning days, and whether feedback on performance is provided (Tinga et al., 2019a). However, when controlling for individual differences and for task-related aspects, effect sizes for measuring learning through neurophysiological outcome measures become as large as effect sizes for measuring learning through behavioral outcome measures. This means that even though the sensitivity of neurophysiology might make assessment somewhat more complicated, neurophysiology can provide just as powerful insights in learning as behavior. Potentially, the sensitivity of neurophysiological outcome measures to individual differences and task-related aspects could even provide for valuable insights into learning processes in addition to insights obtained through behavioral measurements.
Signal Noise and Limitation of Freedom of Movement of the Trainee
Non-invasive neurophysiological measures can be noisy, especially outside of the controlled environment of the lab. Most techniques provide the most valid outcomes under ideal conditions. Commonly encountered noise sources are electrode motion and muscle contraction artifacts, usually caused by movements of the trainee (Vazquez et al., 2012; Poungponsri and Yu, 2013). The sensitivity to motion and muscle activity can be problematic for techniques using electrodes placed on the body, especially when measuring signals of lower amplitudes, such as EEG. When collecting eye-related measures using an eye tracker, muscle and motion artifacts are non-existent as no electrodes are placed on the body of the trainee, while other sources of noise can arise such as movements of the eye tracking camera with respect to the eye and changes in corneal reflection (Li et al., 2008). To deal with artifacts, in research on neurophysiology in learning the trainee is generally presented with a seated task and instructed to remain seated as still as possible. This brings us to another weakness, namely that of the limitation of freedom of movement of the trainee, which can, besides by being induced by the design of the task, be induced by neurophysiological measuring equipment worn by the trainee. Most such equipment is wired, and, depending on the type of measurement collected, can be uncomfortable to wear (Liao et al., 2012), affecting the trainee’s mobility.
Technical Challenges and Personnel Training
Technical challenges also arise when applying neurophysiology in learning and training, as collection, synchronization, and analysis of large amounts of data from multi-channel data streams are required (Dahlstrom-Hakki et al., 2019). Technological advancements are making these requirements increasingly easier, yet overcoming these technical challenges as much as possible is time-consuming as things such as time desynchronization between multiple devices and delays between the device and scripts occur frequently (Dahlstrom-Hakki et al., 2019). Sufficient attention and time should be given to personnel training and device maintenance and extra time should also be allocated for experimental preparations and procedures during testing to ensure that data of high quality is collected (Kivikangas et al., 2011).
Hardware and Software Expenses
Besides the costs associated with the time-investment in personnel training, maintenance of devices and experimental preparations and procedures, hardware, and software for non-invasive neurophysiological measurements can also be expensive. For example, costs of state-of-the-art eye tracking equipment range between US$ 20,000 and US$ 30,000 (Ferhat and Vilariño, 2016) and EEG equipment can cost more than US$ 75,000 (Krigolson et al., 2017). Costs of sensors measuring peripheral physiology such as heart rate, respiration and EDA can also be high (Lucena et al., 2015), with a complete research system including hardware and software easily reaching US$ 10,000 per modality.
Opportunities for Applying Neurophysiological Measures in Learning and Training
Potential Is Recognized and the Field Is Highly Interdisciplinary
Applying non-invasive measures of neurophysiology opens up many opportunities. Multiple scholars have pointed out the potential of neurophysiology in learning and training. For example, Lai et al. (2013) reviewed 113 studies evaluating the effect of learning in general on eye-related outcome measures and indicated that they are promising in assessing learning within various themes with various topics. Regarding research on brain activity in learning, a review of about 90 studies on EEG changes during motor learning (Krigolson et al., 2015) concluded specific event-related brain potentials (ERPs) are able to provide important insights into learning. Additionally, Leff et al. (2011) assessed over 80 studies on brain activity measured via fNIRS during motor tasks and pointed out that this modality is promising for studying motor learning. Tinga et al. (2019a) reviewed 113 experiments examining a wide range of non-invasive neurophysiological measures during learning (including measures of brain activity, eye-related outcome measures, measures of heart rate, skin conductance and respiration) and demonstrated that effect sizes were large for neurophysiological changes during learning, although they were smaller than for behavioral changes. Yet, the difference in effect sizes between neurophysiology and behavior disappeared when controlling for task-related aspects and individual differences. Again, the authors concluded that neurophysiology is promising for the assessment of learning. Although most work on neurophysiology in learning is conducted in the controlled environment of the lab, research has also demonstrated neurophysiological measures can be successfully applied for examining learning in real-life scenarios such as in a classroom (Ko et al., 2017; Poulsen et al., 2017; Bevilacqua et al., 2019; Dahlstrom-Hakki et al., 2019).
The research field of neurophysiology in learning and training is also highly interdisciplinary, which allows for combining knowledge from experts in fields such as sensor technology, signal processing, artificial intelligence, neurophysiology, and engineering. Of course, it is a challenge to oversee and bring together the research in all these areas of expertise (Brouwer et al., 2015), but when one brings together people from these different areas we believe that this will greatly benefit formulating solutions.
Employing General Changes in Neurophysiology
The feasibility of applying measures of neurophysiology outside the lab increases, when examining changes in neurophysiology reflective of general changes in mental effort, instead of examining specific responses to specific stimuli as is the case for example with ERPs from EEG. Specific responses can only be computed when a task itself presents many similar stimuli that can elicit an ERP that can be reliably time-locked to those stimuli, or when a task-irrelevant probe can be repeatedly presented during learning (Gevins and Smith, 2003). Therefore, measures of more general changes in neurophysiology over time that do not need to be precisely time-locked after repeated exposure of a specific event, such as changes in frequency bands in EEG, pupil size or heart rate, are presently better suited for application in real-world learning and training contexts, opening up many possibilities.
Solutions for Dealing With Noise and Real-Time Data Analysis and Visualization
As discussed above, neurophysiological techniques generally provide the most valid outcomes under ideal conditions. Yet, solutions for dealing (automatically) with noise are available and are being enhanced (Vorobyov and Cichocki, 2002; Vazquez et al., 2012; Poungponsri and Yu, 2013), increasing validity and applicability. The rapid developments in machine learning contribute to the processing (e.g., enhancing signal to noise ratio) and analysis (e.g., exploiting multiple modalities) of neurophysiological measures (Müller et al., 2008; Lemm et al., 2011).
Moreover, there are several ways in which neurophysiological data can be analyzed, with some applied studies having proposed a simple form of analysis. For example, in analyzing eye tracking data one can look at the percentage of visual attention allocation across regions of interest in the visual field, which is easier to perform even with eye-trackers with a lower sample rate and facilitates interpretation (Dahlstrom-Hakki et al., 2019). Yet, simplifying analyses can come at the cost of having less deep insight into collected data than more advanced analysis techniques. Advanced analyses are however becoming more accessible through the increased availability of powerful software tools. Examples of relatively frequently used open source software tools for aiding in analyzing brain activity signals are FieldTrip (Oostenveld et al., 2011), Brainstorm (Tadel et al., 2011), and MNE (Gramfort et al., 2013).
Neurophysiological signals can also be analyzed and visualized in real-time. For example, it is possible to analyze EEG data online taking into account individual variations (Guger et al., 2000) and to visualize the data using open-source software (Mullen et al., 2013). Another example is real-time analysis and monitoring of neurophysiology via a cloud system which has shown to be accurate and effective for ECG (Xia et al., 2013). These advancements aid the ease and speed at which data can be assessed and support providing the trainee with effective feedback and tailoring the learning experience to the individual.
Development of Wireless Sensors and Reduction of Costs
Advancements to enhance the trainee’s comfort and ease of movement are also being made with wireless and more compact sensors already being made available. For example, wearable wristbands such as the Empatica E4 and the Everion (Biovotion) are able to measure multiple peripheral neurophysiological outcomes and systems such as B-ALERT (Advanced Brain Monitoring) and g.Nautilus (g.tec) provide wireless EEG measures. These newer devices require considerably less set-up time and are less sensitive to motion artifacts. Sensors are even becoming fully portable with the possibility for data collection to occur via a smartphone or tablet, even for measures of brain activity (Stopczynski et al., 2014; Poulsen et al., 2017). All these devices promise to provide medical grade data and several studies have provided evidence of their successful application (Amaral et al., 2017; Corino et al., 2017; Barrios et al., 2018; Beeler et al., 2018; Tinga et al., 2019b).
In general, wearables and more applied systems are substantially cheaper, easily being 5–20 times less costly to purchase. Together with the reduction of set-up time and advanced solutions for dealing with noise and enhancing data analysis, the total costs involved for being able to apply neurophysiology in learning and training are greatly reduced.
However, the performance of these systems is in many instances still not on par with more traditional lab-based systems due to limitations such as a lower number of channels, less precise application and lower sample rate (Hairston et al., 2014; Melnik et al., 2017). Yet, we expect technological advancements in the upcoming years to substantially reduce the difference in data quality between lab-based systems and wearables, making these wearables increasingly interesting to apply.
Embedding Sensors in Learning Systems and Widely Used Devices
Another interesting direction for future developments is to embed equipment measuring neurophysiology in learning systems. Such systems could then potentially be applied to diagnose learning states and provide instantaneous help or adapt the material based on the recorded neurophysiological data (Lai et al., 2013). One category of systems that has high potential in education and training is virtual reality (de Back et al., submitted) and existing virtual reality headsets and to-be developed headsets with integrated eye tracking and brain activity sensors can contribute to this potential (Jantz et al., 2017). As an example, Sood and Singh (2018) proposed a framework on combining virtual reality and EEG in an educational game for robust and resilient learning with an enhanced quality of the learning experience.
Additionally, integrating sensors in smartphones, laptops, and tablets might also be an interesting endeavor, with for example advancements being made in using cameras of such devices for collecting eye-related measures (Lin et al., 2013; Lim et al., 2015; Krafka et al., 2016; Meißner et al., 2019) and measures of heart rate and heart rate variability (Heathers, 2013; Lucena et al., 2015).
Threats for Applying Neurophysiological Measures in Learning and Training
Measures Are Mostly Applied in Lab Settings Using Controlled Tasks
Although applying neurophysiology in learning and training brings along interesting possibilities, there are important threats involved that also need to be considered. To start with, studies examining neurophysiology in learning are mostly fundamental, being conducted in the controlled environment of the lab using carefully selected tasks with simple stimuli (Dahlstrom-Hakki et al., 2019; Tinga et al., 2019a). This fundamental work is obviously important to understand in what way and under what circumstances neurophysiology provides insight into learning. Yet, if we want to be able to assess and enhance learning in the real-world, outside of the lab, we would benefit from combining this fundamental knowledge with more applied knowledge.
Focus on Specific Learning Topics and Specific Neurophysiological Measures
The majority of studies on non-invasive neurophysiology in learning have focused on brain activity and eye-related outcome measures, while relatively few studies examined other outcome measures such as skin conductance, heart rate, EMG, and respiration (Leff et al., 2011; Lai et al., 2013; Krigolson et al., 2015; Tinga et al., 2019a), making it unclear how well current findings generalize to these other outcome measures. Additionally, most work has focused on perceptual learning and the acquisition of new information or conceptual knowledge (Lai et al., 2013; Tinga et al., 2019a). Some learning topics should be explored in more detail in order to ensure generalizability across a wide range of tasks (Krigolson et al., 2015). Therefore, more research is needed examining a wider range of neurophysiological measures on a range of tasks.
Validity and Applicability of Neurophysiology in Learning and Training
Although many scholars have deemed neurophysiology as having high potential in assessing learning and training, some important concerns about validity and applicability have been raised by another group of scholars. Brouwer et al. (2014) pointed out changes in neurophysiology over time during learning might not be specifically related to the learning process itself, but rather reflect unspecific changes. Brouwer et al. (2014) demonstrated that multiple neurophysiological measures (i.e., EEG, ECG, EDA, respiration, and eye-related measures) were sensitive to difficulty level but that there was no interaction between difficulty level and time on task during learning. Similarly, Cowley (2015) argued neurophysiology is able to measure simple cognitive states, but is limited in measuring higher-order cognition. Ansari et al. (2011) even argue that it is shortsighted and unrealistic to expect that results from research in cognitive neuroscience will have a direct and immediate impact in education. Dahlstrom-Hakki et al. (2019) also indicated that we are still far from being able to use neurophysiology to directly measure learning or knowledge, with these tools at best allowing us to make inferences about a complex process such as learning by studying it during carefully structured tasks. Concerns like these raise the question how valid and applicable these measures are in assessing learning, especially outside the environment of the lab. Indeed, most of the literature measuring neurophysiology during task performance focuses on measuring and predicting simple cognitive states such as arousal or attention, without examining learning (Charles and Nixon, 2019).
Ethical Challenges and Feeling Uncomfortable Being Measured Through Sensors
The collection of a large quantity of psychological data originating from the body of a trainee obviously brings along ethical challenges. Such a large dataset on bodily signals might contain sensitive information in addition to effects of learning. For example, neurological, respiratory, and cardiac disorders are known to influence neurophysiological measures (Willems et al., 1991; Daly and Wolpaw, 2008). Even someone’s physical and psychological health affects neurophysiology (Lovallo, 2011; Iorfino et al., 2016). One needs to ensure that this data will not be used for unethical purposes.
Additionally, trainees might feel uncomfortable being measured through sensors, with the sensors potentially inducing fear. For instance, Neven (2015) discussed the effects of a home monitoring system for the elderly consisting of cameras and sensors. Neven (2015) described that one woman developed “sensor phobia”, an intense fear of sensors. Albeit an extreme example, the fear of sensors alongside the ethical issues pose a threat to applying neurophysiological measures, including those for learning and training.
Non-experts Perception of Warrantability of Neurophysiology
A final threat is one related to the perception of the layman audience. Non-experts tend to regard neurophysiology to convey an objective truth, even if conclusions based on neurophysiology are unfounded. For example, Weisberg et al. (2008) found that a group of non-experts judged explanations of phenomena containing brain-based information as more satisfying than explanations without this information, even when that information was logically irrelevant. Especially bad explanations were rated more satisfying when irrelevant neurophysiological information was added. When it comes to neurophysiology in learning and training specifically, educational concepts based on neurophysiology can be popular even when claims are not backed by scientific evidence (Howard-Jones, 2009). These results indicate that one needs to be careful that non-experts are not misled by explanations using neurophysiological information. Researchers and practitioners should therefore ensure they are clear about what can presently be concluded from their results and what might become possible in the future (Brouwer et al., 2015).
Conclusion
Considering the various aspects of the SWOT analysis, it is apparent that the field of neurophysiology in learning and training is developing rapidly with many opportunities existing and emerging. In these developments, weaknesses should be taken into account, such as the fact that neurophysiology responds to a wide variety of functions supported by the nervous system, and technical challenges related to synchronization and analysis of large amount of data. We should also be aware of the threats that are involved, which includes the fact that research has mainly been conducted in controlled lab environments focusing on specific learning topics and specific neurophysiological measures, as well as ethical challenges and trainees possibly feeling uncomfortable being measured through sensors. Yet, neurophysiology in learning and training has powerful and valuable strengths, such as allowing for formative and objective assessment, tailoring the task to the individual, and gaining deeper insights through multi-channel data. Additionally, there are several opportunities for the field of neurophysiology in learning and training.
To start with, as the associated costs to record neurophysiological measures continue to go down and equipment becomes less intrusive for the trainee and can be set-up more quickly and easily, applying non-invasive neurophysiology in learning and training will become increasingly feasible. We expect technological advancements in the upcoming years to substantially reduce the difference in data quality between lab-based systems and wearables. The advancements will allow for focusing on examining effects in more detail outside of the lab to make sure that current results established in the lab replicate in more applied and less controlled environments. Moreover, it will become more feasible to assess a wider range of measures to add to the literature which is currently mostly focused on measures of brain activity and eye-related outcomes.
The important challenge to distinguish learning-specific effects from other effects will become easier to deal with when research has identified under which conditions effects will hold for which neurophysiological measures, in combination with advancements in data quality and dealing with noise. Especially when taking advantage of the highly interdisciplinary nature of the research field, experts can work together in advancing toward solutions to dealing with challenges like these.
As in any other field, ethical issues such as proper data handling should be considered carefully. These issues will especially be apparent as application of non-invasive neurophysiology in learning increasingly takes place outside the lab. One possible direction could be to take advantage of real-time data analysis and visualization of more general changes in neurophysiology, which does not necessarily involve saving a full dataset with specific, and possibly sensitive, responses. This online assessment of neurophysiological effects could already provide trainers, and even trainees, with important information on the learning process and can be used to tailor the learning task in real-time.
Also, one should ensure findings in the field are communicated clearly and acknowledge limitations in order to warrant that findings are not misinterpreted. Here one could also take advantage of the highly interdisciplinary nature of the research field, which allows for addressing a wide range of aspects to guarantee a comprehensive view of new findings.
Developments in including neurophysiological sensors in learning systems and new technologies such as virtual reality systems will open up many interesting possibilities for application. For example, promising immersive learning experiences can be more effectively adapted to the individual when making use of online (automatic) formative assessment through neurophysiology.
Opportunities of neurophysiology in learning and training can be seized most effectively by having insight into the weaknesses and threats of their application as well as their unique strengths. In this way, the field of non-invasive neurophysiology in learning and training can move forward in promising directions getting closer toward valid application of neurophysiology in a wide range of learning and training settings.
Author Contributions
All authors: conceptualization, and writing-review and editing. AT: writing-original draft preparation. ML: funding acquisition.
Funding
The current research is funded by the European Union, OP Zuid, the Ministry of Economic Affairs, the Province of Noord-Brabant and the municipalities of Tilburg and Gilze Rijen (PROJ-00076) awarded to ML. The usual exculpations apply.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Acharya, U. R., Kannathal, N., Hua, L. M., and Yi, L. M. (2005). Study of heart rate variability signals at sitting and lying postures. J. Bodywork Mov. Ther. 9, 134–141. doi: 10.1016/j.jbmt.2004.04.001
Alain, C., Campeanu, S., and Tremblay, K. (2010). Changes in sensory evoked responses coincide with rapid improvement in speech identification performance. J. Cogn. Neurosci. 22, 392–403. doi: 10.1162/jocn.2009.21279
Alain, C., and Snyder, J. S. (2008). Age-related differences in auditory evoked responses during rapid perceptual learning. Clin. Neurophysiol. 119, 356–366. doi: 10.1016/j.clinph.2007.10.024
Amaral, C. P., Simoes, M. A., Mouga, S., Andrade, J., and Castelo-Branco, M. (2017). A novel brain computer interface for classification of social joint attention in autism and comparison of 3 experimental setups: a feasibility study. J. Neurosci. Methods 290, 105–115. doi: 10.1016/j.jneumeth.2017.07.029
Ansari, D., Coch, D., and De Smedt, B. (2011). Connecting education and cognitive neuroscience: Where will the journey take us? Educ. Philos. Theory 43, 37–42. doi: 10.1111/j.1469-5812.2010.00705.x
Antonenko, P., Paas, F., Grabner, R., and van Gog, T. (2010). Using electroencephalography to measure cognitive load. Educ. Psychol. Rev. 22, 425–438. doi: 10.1007/s10648-010-9130-y
Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., and Onaral, B. (2012). Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 36–47. doi: 10.1016/j.neuroimage.2011.06.023
Backs, R. W., and Seljos, K. A. (1994). Metabolic and cardiorespiratory measures of mental effort: The effects of level of difficulty in a working memory task. Int. J. Psychophysiol. 16, 57–68. doi: 10.1016/0167-8760(94)90042-6
Baillet, S., Mosher, J. C., and Leahy, R. M. (2001). Electromagnetic brain mapping. IEEE Signal Process. Mag. 18, 14–30. doi: 10.1109/79.962275
Barrios, L., Oldrati, P., Santini, S., and Lutterotti, A. (2018). “Recognizing digital biomarkers for fatigue assessment in patients with multiple sclerosis,” in Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare; PervasiveHealth (New York, NY: EAI), doi: 10.4108/eai.20-4-2018.2276340
Barrios, V. M. G., Gütl, C., Preis, A. M., Andrews, K., Pivec, M., Mödritscher, F., et al. (2004). “AdELE: A framework for adaptive e-learning through eye tracking,” in Proceedings of the I-KNOW’04, Graz.
Barron, A. B., Hebets, E. A., Cleland, T. A., Fitzpatrick, C. L., Hauber, M. E., and Stevens, J. R. (2015). Embracing multiple definitions of learning. Trends Neurosci. 38, 405–407. doi: 10.1016/j.tins.2015.04.00
Beeler, N., Roos, L., Delves, S. K., Veenstra, B. J., Friedl, K., Buller, M. J., et al. (2018). The wearing comfort and acceptability of ambulatory physical activity monitoring devices in soldiers. IISE Transact. Occupat. Ergon. Hum. Fact. 6, 1–10. doi: 10.1080/24725838.2018.1435431
Berntson, G. G., and Cacioppo, J. T. (2000). “From homeostasis to allodynamic regulation,” in Handbook of Psychophysiology, 2nd Edn, eds J. T. Cacioppo, L. G. Tassinary, and G. G. Berntson, (New York, NY: Cambridge University Press), 459–481.
Berntson, G. G., Cacioppo, J. T., and Quigley, K. S. (1991). Autonomic determinism: The modes of autonomic control, the doctrine of autonomic space, and the laws of autonomic constraint. Psychol. Rev. 98:459. doi: 10.1037//0033-295x.98.4.459
Bevilacqua, D., Davidesco, I., Wan, L., Chaloner, K., Rowland, J., Ding, M., et al. (2019). Brain-to-brain synchrony and learning outcomes vary by student–teacher dynamics: Evidence from a real-world classroom electroencephalography study. J. Cogn. Neurosci. 31, 401–411. doi: 10.1162/jocn_a_01274
CrossRef Full Text %M 29708820 | Google Scholar
Borghini, G., Arico, P., Graziani, I., Salinari, S., Sun, Y., Taya, F., et al. (2016). Quantitative assessment of the training improvement in a motor-cognitive task by using EEG, ECG and EOG signals. Brain Topogr. 29, 149–161. doi: 10.1007/s10548-015-0425-7
Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., and Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75. doi: 10.1016/j.neubiorev.2012.10.003
Brehm, J. W., and Self, E. A. (1989). The intensity of motivation. Ann. Rev. Psychol. 40, 109–131. doi: 10.1146/annurev.ps.40.020189.000545
Brenner, E., Chirculescu, A. R. M., Reblet, C., and Smith, C. (2015). Assessment in anatomy. Eur. J. Anat. 19, 105–124.
Brouwer, A. M., Hogervorst, M. A., Holewijn, M., and van Erp, J. B. (2014). Evidence for effects of task difficulty but not learning on neurophysiological variables associated with effort. Int. J. Psychophysiol. 93, 242–252. doi: 10.1016/j.ijpsycho.2014.05.004
Brouwer, A. M., Hogervorst, M. A., Van Erp, J. B., Heffelaar, T., Zimmerman, P. H., and Oostenveld, R. (2012). Estimating workload using EEG spectral power and ERPs in the N-back task. J. Neur. Eng. 9:045008. doi: 10.1088/1741-2560/9/4/045008
Brouwer, A. M., Zander, T. O., van Erp, J. B., Korteling, J. E., and Bronkhorst, A. W. (2015). Using neurophysiological signals that reflect cognitive or affective state: Six recommendations to avoid common pitfalls. Front. Neurosci. 9:136. doi: 10.3389/fnins.2015.00136
Charles, R. L., and Nixon, J. (2019). Measuring mental workload using physiological measures: a systematic review. Appl. Ergon. 74, 221–232. doi: 10.1016/j.apergo.2018.08.028
Corino, V. D. A., Laureanti, R., Ferranti, L., Scarpini, G., Lombardi, F., and Mainardi, L. T. (2017). Detection of atrial fibrillation episodes using a wristband device. Physiol. Measur. 38, 787–799. doi: 10.1088/1361-6579/aa5dd7
Cowley, B. (2015). Psychophysiology and high-performance cognition - a brief review of the literature. PeerJ 3:e1373v1. doi: 10.7287/peerj.preprints.1373v1
Cowley, B., Ravaja, N., and Heikura, T. (2013). Cardiovascular physiology predicts learning effects in a serious game activity. Comput. Educ. 60, 299–309. doi: 10.1016/j.compedu.2012.07.014
Crosson, B., Ford, A., McGregor, K. M., Meinzer, M., Cheshkov, S., Li, X., et al. (2010). Functional imaging and related techniques: an introduction for rehabilitation researchers. J. Rehab. Res. Dev. 47, vii–xxxiv. doi: 10.1682/JRRD.2010.02.0017
Dahlstrom-Hakki, I., Asbell-Clarke, J., and Rowe, E. (2019). Showing is knowing: The potential and challenges of using neurocognitive measures of implicit learning in the classroom. Mind Brain Educ. 13, 30–40. doi: 10.1111/mbe.12177
Daly, J. J., and Wolpaw, J. R. (2008). Brain–computer interfaces in neurological rehabilitation. Lancet Neurol. 7, 1032–1043. doi: 10.1016/S1474-4422(08)70223-0
Díaz, D. H. ÓC., and Pallas-Areny, R. (2010). “Heart rate detection from single-foot plantar bioimpedance measurements in a weighing scale,” in Annual International Conference of the IEEE Engineering in Medicine and Biology (Piscataway, NJ: IEEE), doi: 10.1109/IEMBS.2010.5627358
Eckstein, M. K., Guerra-Carrillo, B., Miller Singley, A. T., and Bunge, S. A. (2017). Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? Dev. Cogn. Neurosci. 25, 69–91. doi: 10.1016/j.dcn.2016.11.001
Eppinger, B., and Kray, J. (2011). To choose or to avoid: Age differences in learning from positive and negative feedback. J. Cogn. Neurosci. 23, 41–52. doi: 10.1162/jocn.2009.21364
Eppinger, B., Kray, J., Mock, B., and Mecklinger, A. (2008). Better or worse than expected? Aging, learning, and the ERN. Neuropsychologia 46, 521–539. doi: 10.1016/j.neuropsychologia.2007.09.001
Fairclough, S. H., and Ewing, K. (2017). The effect of task demand and incentive on neurophysiological and cardiovascular markers of effort. Int. J. Psychophysiol. 119, 58–66. doi: 10.1016/j.ijpsycho.2017.01.007
Fairclough, S. H., Venables, L., and Tattersall, A. (2005). The influence of task demand and learning on the psychophysiological response. Int. J. Psychophysiol. 56, 171–184. doi: 10.1016/j.ijpsycho.2004.11.003
Ferhat, O., and Vilariño, F. (2016). Low cost eye tracking: the current panorama. Comput. Intell. Neurosci. 14:8680541. doi: 10.1155/2016/8680541
Fowles, D. C., Christie, M. J., Edelberg, R., Grings, W. W., Lykken, D. T., and Venables, P. H. (1981). Publication recommendations for electrodermal measurements. Psychophysiology 18, 232–239. doi: 10.1111/j.1469-8986.1981.tb03024.x
Francis, J. (2016). ECG monitoring leads and special leads. Indian Pacing Electrophysiol. J. 16, 92–95. doi: 10.1016/j.ipej.2016.07.003
Gable, T. M., Kun, A. L., Walker, B. N., and Winton, R. J. (2015). “Comparing heart rate and pupil size as objective measures of workload in the driving context,” in Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Nottingham, doi: 10.1145/2809730.2809745
Gergelyfi, M., Jacob, B., Olivier, E., and Zenon, A. (2015). Dissociation between mental fatigue and motivational state during prolonged mental activity. Front. Behav. Neurosci. 9:176. doi: 10.3389/fnbeh.2015.00176
Gevins, A., and Smith, M. E. (2003). Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci. 4, 113–131. doi: 10.1080/14639220210159717
Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., et al. (2013). MEG and EEG data analysis with MNE-Python. Front. Neurosci. 7:267. doi: 10.3389/fnins.2013.00267
Guger, C., Ramoser, H., and Pfurtscheller, G. (2000). Real-time EEG analysis with subject-specific spatial patterns for a brain-computer interface (BCI). IEEE Transact. Rehabil. Eng. 8, 447–456. doi: 10.1109/86.895947
Hairston, W. D., Whitaker, K. W., Ries, A. J., Vettel, J. M., Bradford, J. C., Kerick, S. E., et al. (2014). Usability of four commercially-oriented EEG systems. J. Neural Eng. 11:14. doi: 10.1088/1741-2560/11/4/046018
Hattie, J., and Timperley, H. (2007). The power of feedback. Rev. Educ. Res. 77, 81–112. doi: 10.3102/003465430298487
Heathers, J. A. J. (2013). Smartphone-enabled pulse rate variability: an alternative methodology for the collection of heart rate variability in psychophysiological research. Int. J. Psychophysiol. 89, 297–304. doi: 10.1016/j.ijpsycho.2013.05.017
Hiyamizu, M., Maeoka, H., Matsuo, A., and Morioka, S. (2014). Effects of self-action observation on standing balance learning: A change of brain activity detected using functional near-infrared spectroscopy. Neurorehabilitation 35, 579–585. doi: 10.3233/nre-141153
Hockey, G. R. J. (1997). Compensatory control in the regulation of human performance under stress and high workload: a cognitive-energetical framework. Biol. Psychol. 45, 73–93. doi: 10.1016/S0301-0511(96)05223-4
Höfle, M., Kenntner-Mabiala, R., Pauli, P., and Alpers, G. W. (2008). You can see pain in the eye: Pupillometry as an index of pain intensity under different luminance conditions. Int. J. Psychophysiol. 70, 171–175. doi: 10.1016/j.ijpsycho.2008.06.008
Hogervorst, M. A., Brouwer, A. M., and van Erp, J. B. (2014). Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Front. Neurosci. 8:322. doi: 10.3389/fnins.2014.00322
Howard-Jones, P. A. (2009). Scepticism is not enough. Cortex 45, 550–551. doi: 10.1016/j.cortex.2008.06.002
Iorfino, F., Hickie, I. B., Lee, R. S. C., Lagopoulos, J., and Hermens, D. F. (2016). The underlying neurobiology of key functional domains in young people with mood and anxiety disorders: A systematic review. BMC Psychiatry 16:156. doi: 10.1186/s12888-016-0852-3
Jantz, J., Molnar, A., and Alcaide, R. (2017). “A brain-computer interface for extended reality interfaces,” in Proceedings of the ACM SIGGRAPH 2017 VR Village, Los Angeles, CA, 1–2. doi: 10.1145/3089269.3089290
Jarchi, D., and Casson, A. J. (2016). “Estimation of heart rate from foot worn photoplethysmography sensors during fast bike exercise,” in Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Piscataway, NJ: IEEE), doi: 10.1109/EMBC.2016.7591398
Kivikangas, J. M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., et al. (2011). A review of the use of psychophysiological methods in game research. J. Gaming Virt. Worlds 3, 181–199. doi: 10.1386/jgvw.3.3.181_1
Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Rev. 29, 169–195. doi: 10.1016/s0165-0173(98)00056-3
Klimesch, W., Schack, B., and Sauseng, P. (2005). The functional significance of theta and upper alpha oscillations. Exp. Psychol. 52, 99–108. doi: 10.1027/1618-3169.52.2.99
Ko, L.-W., Komarov, O., Hairston, W. D., Jung, T.-P., and Lin, C.-T. (2017). Sustained attention in real classroom settings: an EEG study. Front. Hum. Neurosci. 11:388. doi: 10.3389/fnhum.2017.00388
Kopton, I. M., and Kenning, P. (2014). Near-infrared spectroscopy (NIRS) as a new tool for neuroeconomic research. Front. Hum. Neurosci. 8:549. doi: 10.3389/fnhum.2014.00549
Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., et al. (2016). “Eye tracking for everyone,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Piscataway, NJ: IEEE), 2176–2184. doi: 10.1109/cvpr.2016.239
Krigolson, O. E., Cheng, D. R., and Binsted, G. (2015). The role of visual processing in motor learning and control: Insights from electroencephalography. Vis. Res. 110, 277–285. doi: 10.1016/j.visres.2014.12.024
Krigolson, O. E., Pierce, L. J., Holroyd, C. B., and Tanaka, J. W. (2009). Learning to become an expert: Reinforcement learning and the acquisition of perceptual expertise. J. Cogn. Neurosci. 21, 1833–1840. doi: 10.1162/jocn.2009.21128
Krigolson, O. E., Williams, C. C., Norton, A., Hassall, C. D., and Colino, F. L. (2017). Choosing muse: Validation of a low-cost, portable EEG system for ERP research. Front. Neurosci. 11:109. doi: 10.3389/fnins.2017.00109
Kwon, O., Jeong, J., Kim, H. B., Kwon, I. H., Park, S. Y., Kim, J. E., et al. (2018). Electrocardiogram sampling frequency range acceptable for heart rate variability analysis. Healthcare Inform. Res. 24, 198–206. doi: 10.4258/hir.2018.24.3.198
Lai, M. L., Tsai, M. J., Yang, F. Y., Hsu, C. Y., Liu, T. C., Lee, S. W. Y., et al. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educ. Res. Rev. 10, 90–115. doi: 10.1016/j.edurev.2013.10.001
Larsen, R. S., and Waters, J. (2018). Neuromodulatory correlates of pupil dilation. Front. Neural Circ. 12:21. doi: 10.3389/fncir.2018.00021
Leff, D. R., Orihuela-Espina, F., Elwell, C. E., Athanasiou, T., Delpy, D. T., Darzi, A. W., et al. (2011). Assessment of the cerebral cortex during motor task behaviours in adults: a systematic review of functional near infrared spectroscopy (fNIRS) studies. Neuroimage 54, 2922–2936. doi: 10.1016/j.neuroimage.2010.10.058
Lemm, S., Blankertz, B., Dickhaus, T., and Müller, K. R. (2011). Introduction to machine learning for brain imaging. Neuroimage 56, 387–399. doi: 10.1016/j.neuroimage.2010.11.004
León-Carrión, J., and León-Domínguez, U. (2012). “Functional near-infrared spectroscopy (fNIRS): Principles and neuroscientific applications,” in Neuroimaging-Methods, ed. P. Bright, (London: InTech), doi: 10.5772/23146
Li, F., Munn, S., and Pelz, J. (2008). A model-based approach to video-based eye tracking. J. Modern Optics 55, 503–531. doi: 10.1080/09500340701467827
Liao, L. D., Chen, C. Y., Wang, I. J., Chen, S. F., Li, S. Y., Chen, B. W., et al. (2012). Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors. J. Neuroeng. Rehabil. 9:11. doi: 10.1186/1743-0003-9-5
Lim, J. H., Lee, J. H., Hwang, H. J., Kim, D. H., and Im, C. H. (2015). Development of a hybrid mental spelling system combining SSVEP-based brain-computer interface and webcam-based eye tracking. Biomed. Signal Process. Control 21, 99–104. doi: 10.1016/j.bspc.2015.05.012
Lin, Y. T., Lin, R. Y., Lin, Y. C., and Lee, G. C. (2013). Real-time eye-gaze estimation using a low-resolution webcam. Multimed. Tools Appl. 65, 543–568. doi: 10.1007/s11042-012-1202-1
Lovallo, W. R. (2011). Do low levels of stress reactivity signal poor states of health? Biol. Psychol. 86, 121–128. doi: 10.1016/j.biopsycho.2010.01.006
Lu, G., Yang, F., Taylor, J., and Stein, J. (2009). A comparison of photoplethysmography and ECG recording to analyse heart rate variability in healthy subjects. J. Med. Eng. Technol. 33, 634–641. doi: 10.3109/03091900903150998
Lucena, S. E. D., Sampaio, D. J. B. S., Mall, B., Meyer, M., Burkart, M. A., and Keller, F. V. (2015). “ECG monitoring using android mobile phone and bluetooth,” in IEEE International Instrumentation and Measurement Technology Conference, (Piscataway, NJ: IEEE), doi: 10.1109/I2MTC.2015.7151585
Luft, A. R., and Buitrago, M. M. (2005). Stages of motor skill learning. Mol. Neurobiol. 32:205. doi: 10.1385/mn:32:3:205
Mak, J. N., Chan, R. H. M., and Wong, S. W. H. (2013). “Evaluation of mental workload in visual-motor task: Spectral analysis of single-channel frontal EEG,” in Proceedings of the 39th Annual Conference of the IEEE Industrial Electronics Society, (Piscataway, NJ: IEEE), 8426–8430. doi: 10.1109/iecon.2013.6700546
Mauss, I., and Robinson, M. (2009). Measures of emotion: a review. Cogn. Emot. 23, 209–237. doi: 10.1080/02699930802204677
McDougal, D. H., and Gamlin, P. D. (2015). Autonomic control of the eye. Comprehens. Physiol. 5, 439–473. doi: 10.1002/cphy.c140014
Meißner, M., Pfeiffer, J., Pfeiffer, T., and Oppewal, H. (2019). Combining virtual reality and mobile eye tracking to provide a naturalistic experimental environment for shopper research. J. Bus. Res. 100, 445–458. doi: 10.1016/j.jbusres.2017.09.028
Melnik, A., Legkov, P., Izdebski, K., Karcher, S. M., Hairston, W. D., Ferris, D. P., et al. (2017). Systems, subjects, sessions: to what extent do these factors influence EEG data? Front. Hum. Neurosci. 11:150. doi: 10.3389/fnhum.2017.00150
Michel, C. M., and Murray, M. M. (2012). Towards the utilization of EEG as a brain imaging tool. Neuroimage 61, 371–385. doi: 10.1016/j.neuroimage.2011.12.039
Michel, C. M., Murray, M. M., Lantz, G., Gonzalez, S., Spinelli, L., and Grave et al. (2004). EEG source imaging. Clin. Neurophysiol. 115, 2195–2222. doi: 10.1016/j.clinph.2004.06.001
Moisello, C., Meziane, H. B., Kelly, S., Perfetti, B., Kvint, S., Voutsinas, N., et al. (2013). Neural activations during visual sequence learning leave a trace in post-training spontaneous EEG. PLoS One 8:65882. doi: 10.1371/journal.pone.0065882
Mullen, T., Kothe, C., Chi, Y. M., Ojeda, A., Kerth, T., Makeig, S., et al. (2013). “Real-time modeling and 3D visualization of source dynamics and connectivity using wearable EEG,” in Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Piscataway, NJ: IEEE), doi: 10.1109/EMBC.2013.6609968
Müller, K. R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G., and Blankertz, B. (2008). Machine learning for real-time single-trial EEG-analysis: From brain–computer interfacing to mental state monitoring. J. Neurosci. Methods 167, 82–90. doi: 10.1016/j.jneumeth.2007.09.022
Murre, J. M. J. (2014). S-shaped learning curves. Psychonom. Bull. Rev. 21, 344–356. doi: 10.3758/s13423-013-0522-0
Neven, L. (2015). By any means? Questioning the link between gerontechnological innovation and older people’s wish to live at home. Technol. Forecas. Soc. Change 93, 32–43. doi: 10.1016/j.techfore.2014.04.016
Nunan, D., Sandercock, G. R., and Brodie, D. A. (2010). A quantitative systematic review of normal values for short-term heart rate variability in healthy adults. Pacing Clin. Electrophysiol. 33, 1407–1417. doi: 10.1111/j.1540-8159.2010.02841.x
Oostenveld, R., Fries, P., Maris, E., and Schoffelen, J.-M. (2011). Fieldtrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci. 2011:156869. doi: 10.1155/2011/156869
Page, E. B., Webb, E. J., Campbell, D. T., Schwartz, R. D., and Sechrest, L. (1966). Unobtrusive measures: nonreactive research in the social sciences. Am. Educ. Res. J. 3:317. doi: 10.2307/1162043
Pietschmann, M., Endrass, T., and Kathmann, N. (2011). Age-related alterations in performance monitoring during and after learning. Neurobiol. Aging 32, 1320–1330. doi: 10.1016/j.neurobiolaging.2009.07.016
Pietschmann, M., Simon, K., Endrass, T., and Kathmann, N. (2008). Changes of performance monitoring with learning in older and younger adults. Psychophysiology 45, 559–568. doi: 10.1111/j.1469-8986.2008.00651.x
Poulsen, A. T., Kamronn, S., Dmochowski, J., Parra, L. C., and Hansen, L. K. (2017). EEG in the classroom: Synchronised neural recordings during video presentation. Sci. Rep. 7:43916. doi: 10.1038/srep43916
Poungponsri, S., and Yu, X. H. (2013). An adaptive filtering approach for electrocardiogram (ECG) signal noise reduction using neural networks. Neurocomputing 117, 206–213. doi: 10.1016/j.neucom.2013.02.010
Reiner, M., and Gelfeld, T. M. (2014). Estimating mental workload through event-related fluctuations of pupil area during a task in a virtual world. Int. J. Psychophysiol. 93, 38–44. doi: 10.1016/j.ijpsycho.2013.11.002
Reis, P. M., Hebenstreit, F., Gabsteiger, F., von Tscharner, V., and Lochmann, M. (2014). Methodological aspects of EEG and body dynamics measurements during motion. Front. Hum. Neurosci. 8:156. doi: 10.3389/fnhum.2014.00156
Richter, M., Friedrich, A., and Gendolla, G. H. E. (2008). Task difficulty effects on cardiac activity. Psychophysiology 45, 869–875. doi: 10.1111/j.1469-8986.2008.00688.x
Richter, M., Gendolla, G. H. E., and Wright, R. A. (2016). “Three decades of research on motivational intensity theory: What we have learned about effort and what we still don’t know,” in Advances in Motivation Science, Vol 3, A. J. Elliot (Amsterdam: Elsevier)149–186. doi: 10.1016/bs.adms.2016.02.001
Ritter, F. E., and Schooler, L. J. (2001). “The learning curve,” in International Encyclopedia Of The Social & Behavioral Sciences, eds W. Kintch, N. Smelser, and P. Baltes, (Oxford: Elsevier Ltd), 8602–8605. doi: 10.1016/B0-08-043076-7/01480-7
Rizzo, A., and Kim, G. J. (2005). A SWOT analysis of the field of virtual reality rehabilitation and therapy. Pres. Teleoper. Virt. Environ. 14, 119–146. doi: 10.1162/1054746053967094
Roatta, S., and Passatore, M. (2008). “Autonomic effects on skeletal muscle,” in Encyclopedia of Neuroscience, eds M. D. Binder, N. Hirokawa, and U. Windhorst, (Berlin: Springer), doi: 10.1007/978-3-540-29678-2_474
Ruger, M., Gordijn, M. C. M., Beersma, D. G. M., de Vries, B., and Daan, S. (2006). Time-of-day-dependent effects of bright light exposure on human psychophysiology: comparison of daytime and nighttime exposure. Am. J. Physiol. Regul. Int. Compar. Physiol. 290, R1413–R1420. doi: 10.1152/ajpregu.00121.2005
Schneider, W., and Chein, J. M. (2003). Controlled & automatic processing: Behavior, theory, and biological mechanisms. Cogn. Sci. 27, 525–559. doi: 10.1207/s15516709cog2703_8
Schneider, W., and Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychol. Rev. 84, 1–66. doi: 10.1037//0033-295x.84.1.1
Schnotz, W., and Kurschner, C. (2007). A reconsideration of cognitive load theory. Educ. Psychol. Rev. 19, 469–508. doi: 10.1007/s10648-007-9053-4
Setz, C., Arnrich, B., Schumm, J., Marca, R. L., Tröster, G., and Ehlert, U. (2010). Discriminating stress from cognitive load using a wearable EDA device. IEEE Transact. Inform. Technol. Biomed. 14, 410–417. doi: 10.1109/TITB.2009.2036164
Sigrist, R., Rauter, G., Riener, R., and Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psych. Bull. Rev. 20, 21–53. doi: 10.3758/s13423-012-0333-8
Song, J. C., Nam, S., Song, K.-S., and Park, S. Y. (2015). Development of a personalized learning system using gaze tracking system. WSEAS Tranact. Comput. 14, 264–271.
Sood, S. K., and Singh, K. D. (2018). An optical-fog assisted EEG-based virtual reality framework for enhancing e-learning through educational games. Comput. Appl. Eng. Educ. 26, 1565–1576. doi: 10.1002/cae.21965
Stobart, G. (2008). Testing Times: The Uses And Abuses Of Assessment. Abingdon: Routledge, doi: 10.4324/9780203930502
Stoianov, I., Genovesio, A., and Pezzulo, G. (2016). Prefrontal goal codes emerge as latent states in probabilistic value learning. J. Cogn. Neurosci. 28, 140–157. doi: 10.1162/jocn_a_00886
Stopczynski, A., Stahlhut, C., Larsen, J. E., Petersen, M. K., and Hansen, L. K. (2014). The smartphone brain scanner: A portable real-time neuroimaging system. PLoS One 9:e86733. doi: 10.1371/journal.pone.0086733
Tadel, F., Baillet, S., Mosher, J. C., Pantazis, D., and Leahy, R. M. (2011). Brainstorm: a user-friendly application for MEG/EEG analysis. Comput. Intell. Neurosci. 2011:879716. doi: 10.1155/2011/879716
Takeuchi, T., Puntous, T., Tuladhar, A., Yoshimoto, S., and Shirama, A. (2011). Estimation of mental effort in learning visual search by measuring pupil response. PLoS One 6:5. doi: 10.1371/journal.pone.0021973
Tan, H. L., Jenkinson, N., and Brown, P. (2014). Dynamic neural correlates of motor error monitoring and adaptation during trial-to-trial learning. J. Neurosci. 34, 5678–5688. doi: 10.1523/jneurosci.4739-13.2014
Tinga, A. M., de Back, T. T., and Louwerse, M. M. (2019a). Non-invasive neurophysiological measures of learning: a meta-analysis. Neurosci. Biobehav. Rev. 99, 59–89. doi: 10.1016/j.neubiorev.2019.02.001
Tinga, A. M., de Back, T. T., and Louwerse, M. M. (2020). Neurophysiological changes in visuomotor sequence learning provide insight in general learning processes: Measures of brain activity, skin conductance, heart rate and respiration. Int. J. Psychophysiol. 115, 40–48. doi: 10.1016/j.ijpsycho.2020.02.015
Tinga, A. M., Nyklicek, I., Jansen, M. P., de Back, T. T., and Louwerse, M. M. (2019b). Respiratory biofeedback does not facilitate lowering arousal in meditation through virtual reality. Appl. Psychophysiol. Biofeedback 44, 51–59. doi: 10.1007/s10484-018-9421-5
Vazquez, R. R., Velez-Perez, H., Ranta, R., Dorr, V. L., Maquin, D., and Maillard, L. (2012). Blind source separation, wavelet denoising and discriminant analysis for EEG artefacts and noise cancelling. Biomed. Signal Process. Control 7, 389–400. doi: 10.1016/j.bspc.2011.06.005
Vecchio, F., Miraglia, F., and Rossini, P. M. (2019). Tracking neuronal connectivity from electric brain signals to predict performance. Neuroscientist 25, 86–93. doi: 10.1177/1073858418776891
Vorobyov, S., and Cichocki, A. (2002). Blind noise reduction for multisensory signals using ICA and subspace filtering, with application to EEG analysis. Biol. Cybern. 86, 293–303. doi: 10.1007/s00422-001-0298-6
Walter, C., Rosenstiel, W., Bogdan, M., Gerjets, P., and Spüler, M. (2017). Online EEG-based workload adaptation of an arithmetic learning environment. Front. Hum. Neurosci. 11:286–286. doi: 10.3389/fnhum.2017.00286
Wang, Y., Song, Y., Qu, Z., and Ding, Y. (2010). Task difficulty modulates electrophysiological correlates of perceptual learning. Int. J. Psychophysiol. 75, 234–240. doi: 10.1016/j.ijpsycho.2009.11.006
Weinstein, I. (2001). Testing multiple intelligences: Comparing evaluation by simulation and written exam. Clin. Law Rev. 8, 247–286.
Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., and Gray, J. R. (2008). The seductive allure of neuroscience explanations. J. Cogn. Neurosci. 20, 470–477. doi: 10.1162/jocn.2008.20040
Willems, J. L., Abreu-Lima, C., Arnaud, P., van Bemmel, J. H., Brohet, C., Degani, R., et al. (1991). The diagnostic performance of computer programs for the interpretation of electrocardiograms. New Engl. J. Med. 325, 1767–1773. doi: 10.1056/nejm199112193252503
Wilson, G. F., and Russell, C. A. (2003). Operator functional state classification using multiple psychophysiological features in an air traffic control task. Hum. Fact. 45, 381–389. doi: 10.1518/hfes.45.3.381.27252
Wilson, G. F., and Russell, C. A. (2007). Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Hum. Fact. 49, 1005–1018. doi: 10.1518/001872007X249875
Xia, H., Asif, I., and Zhao, X. (2013). Cloud-ECG for real time ECG monitoring and analysis. Comput. Methods Progr. Biomed. 110, 253–259. doi: 10.1016/j.cmpb.2012.11.008
Keywords: learning, training, neurophysiology, brain activity, heart rate, eye tracking, skin conductance, respiration
Citation: Tinga AM, de Back TT and Louwerse MM (2020) Non-invasive Neurophysiology in Learning and Training: Mechanisms and a SWOT Analysis. Front. Neurosci. 14:589. doi: 10.3389/fnins.2020.00589
Received: 16 December 2019; Accepted: 13 May 2020;
Published: 05 June 2020.
Edited by:
Nathalie Just, INRA Centre Val de Loire, FranceReviewed by:
Daniel San Juan Orta, Instituto Nacional de Neurología y Neurocirugía Manuel Velasco Suárez, MexicoEphrem Takele Zewdie, University of Calgary, Canada
Mark Stecker, Independent Practitioner, Fresno, United States
Copyright © 2020 Tinga, de Back and Louwerse. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Angelica M. Tinga, A.M.Tinga@uvt.nl