A correction has been applied to this article in:
Corrigendum: A Review of Possible EEG Markers of Abstraction, Attentiveness and Memorisation in Cyber-Physical Systems for Special Education
Maya Dimitrova
Hiroaki Wagatsuma
Aleksandar Krastev1†- 1Department of Interactive Robotics and Control Systems, Institute of Robotics, Bulgarian Academy of Sciences, Sofia, Bulgaria
- 2Department of Human Intelligence Systems, Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology (KYUTECH), Kitakyushu, Japan
- 3Human-Machines Interaction (HUMAIN) Lab, Department of Computer Science, International Hellenic University (IHU), Kavala, Greece
- 4Engineering School of Gipuzkoa—Eibar Section, Department of Applied Mathematics, University of Basque Country (UPV/EHU), Bilbao, Spain
Cyber-physical systems (CPSs) for special education rely on effective mental and brain processing during the lesson, performed with the assistance of humanoid robots. The improved diagnostic ability of the CPS is a prerogative of the system for efficient technological support of the pedagogical process. The article focuses on the available knowledge of possible EEG markers of abstraction, attentiveness, and memorisation (in some cases combined with eye tracking) related to predicting effective mental and brain processing during the lesson. The role of processing abstraction is emphasised as the learning mechanism, which is given priority over the other mechanisms by the cognitive system. The main markers in focus are P1, N170, Novelty P3, RewP, N400, and P600. The description of the effects is accompanied by the analysis of some implications for the design of novel educational scenarios in inclusive classes.
Introduction
A process of radical transformation of present-day schools is taking place, forcefully accelerated by the recent pandemic. On the one hand, the transformation process is based on the “inclusive class” paradigm, where all children share a common learning “ecosystem” of merging ordinary and special education methods. On the other hand, technology is mediating the process and posing additional demands on the cognitive exchange of ideas, knowledge, attention focus, etc., during classroom learning. Within this ongoing, seemingly chaotic transformation of the nature of the school, the question remains: What is the most essential characteristic of a school, defining it as a necessary and sufficient condition? On a very abstract level, it can be said that a school, by definition, is a “teacher and a bunch of kids”. All other accessories of the teaching process, such as textbooks, laptops, blended learning environments, are supplementary to the main figures “the teacher and one or more learners”. The main role of the teacher is to convey new knowledge to the learner as education is a socially mediated process (e.g., Lieberman, 2007). And, when the human teacher is assisted by a humanoid robot, education will be more efficient than today, provided the robot’s activities resemble the roles and functions of the educator, personally conveying, in a social way, new knowledge to the learner, according to the learner’s individual style and cognitive needs (e.g. Dimitrova and Wagatsuma, 2015; Dimitrova et al., 2020). Detecting and diagnosing mental states of the learners, however, is a challenge to the overall approach yet a realistic aim (e.g., Schartner et al., 2015).
It is essential to substantiate the main claims for expected cognitive advantage for the learner with recent neuro-cognitive studies, while demonstrating the influence of the new technology on the learner well-being. In this, two important aspects are outlined—the human–teacher resemblance1 (HTD) of the robot and the aspect of special education (SED) in the classroom setting of the inclusive class. The analysed EEG and eye-tracking experiments support theories that help in drawing conclusions, which are relevant to the inclusion of both dimensions in the learning scenarios.
Learning Processes Revealed by Observing EEG Activations
Several important brain activation effects are described in the present article within a theoretical approach of analysing the ability of the human brain to build probability distributions of the subjective influences of stimulus events as in Lindskog et al. (2021). This theory has a long-term tradition within psychometrics—more specifically, in unidimensional and multidimensional scaling (see Thurstone, 1927; Torgerson, 1958)—yet relatively new in the analysis of the underlying neuronal activity of mental states. More specifically, “Thurstone’s model assumes that an option’s quality is a Gaussian random variable” (Tsukida and Gupta, 2011, p. 3), where “option” is understood as any stimulus, possessing a “quality”, being assessed by the human. The “quality” is the dimension of the subjective effect, exerted on the subject by the perceived stimulus quality, for example, the colour of an object, the pitch of a sound, the duration of an event, impression of a painting, difficulty of a task, etc. The figures below are in line with the mathematical assumptions of these models and represent probability distributions of psychological effects, resulting from the discussed underlying brain mechanisms. Typically, the form of this probability distribution function is “bell” shaped, that is, a normal distribution of the stimulus effect on the cognitive system is assumed with all formal attributes—mean and standard deviation tailored to the specific stimulus effect (McLeod, 2019). The actual scale values are determined empirically or experimentally (Torgerson, 1958).
P1—Response to Abstraction and the Role of Involuntary Attention
A point of theoretical departure in the present analysis—in relation to HTR—originates from an important study, reported a decade ago (Dubal et al., 2011). It demonstrated that early P1 wave was enhanced on the ten most posterior parietal–occipital electrode sites for emotional compared to neutral facial expressions, irrespective of the face mode—human or machine. Figure 1 presents the International 10-20 System for EEG-MCN (Modified Combinatorial Nomenclature) (Oxley, 2020). The main sections of the EEG array, denoting the respective brain areas, are given by capital letters as follows: F for frontal, P for parietal, O for occipital, T for temporal, and C for central. The odd numbers denote electrode placements on the left and the even numbers on the right; NASION is the frontal midpoint between the eyes, and INION is the end point at the back of the head (Schleiger et al., 2014).2
FIGURE 1
FIGURE 1. International 10-20 System for EEG-MCN by Oxley, B.C. (2020)2.
According to the study by Dubal et al. (2011), the human brain detects the facial expression first, determines whether it expresses an emotion or neutrality, and only after that decides whether the face belongs to a human or a machine. This finding reveals a radical new understanding of the mechanisms of learning by the “social brain” (Dunbar, 2003), where more significant—in an evolutionary sense—are the most recent, social mechanisms, than the earlier perceptual saliency–based detection mechanisms (Lee et al., 2002; Elsherif et al., 2017). Therefore, a significant conceptual assumption can be made in the present context that the cognitive system prioritises information along with the ability to discriminate levels of abstraction, where the more abstract items are being processed earlier than the more concrete ones.
This is in line with the categorical discrimination theory of Eleanor Rosch (Rosch and Mervis, 1975; Rosh, 1987). According to the theory, categories are being compared and subsequently discriminated along probabilistic dimensions, where the most central elements are the closest to the “feature family”, whereas at the outskirts are the rare and less characteristic features of a category. Within a similar probabilistic framework, Moore (2012) explained the emergence of the uncanny valley phenomenon (Mori, 1970; MacDorman, 2005), when people are initially attracted, but then repulsed at the encounter of robots, or androids, sharing “realistic” features with the humans—soft, but cold skin, staring eyes, etc. The effect emerges at the intersection of the broad distribution of possible “non-human” features (Category 1), and the narrower distribution of possible “human” features (Category 2), yet dependent on the internal subjective threshold within the overlapping “non-human” part onto the “human” area. Figure 2 is a representation of the conceptualisation of the process of comparison of two categories.
FIGURE 2
FIGURE 2. Conceptualisation of the process of subjective comparison of two categories.
It was shown that the so-called “perceptual magnet effect” is also present in understanding robot utterances by the human (e.g., Read and Belpaeme, 2016), which, again, favours the position toward endowing humanoid robots with abstract “teacher assistant” abilities. Evidently, the effect of the uncanny valley emerges at the “clash” of abstract human-like features and the presence of concrete features like “cold skin” (which should be avoided in robot design). Therefore, the human brain responds to the abstraction with higher priority than to the set of immediate/concrete perceptual features. This ability makes possible to learn that a penguin is a bird and a bat is a mammal without any internal confusion. Therefore, we believe that robots can resemble human teachers in their deep, abstract features, irrespective of the appearances. Moreover, what the Dubal et al. study had revealed, this holds on the emotional level too. A robot’s display of emotions is, therefore, equally well accepted by the human learner as is the emotional reaction of the human teacher.
Extensive reviews of the EEG marker of involuntary attention called Mismatch Negativity (MMN), which is typically elicited between 100 and 150 ms in the fronto-central areas of the brain after the onset of a typically auditory stimulus, which is somewhat surprising or different, are provided in Duncan et al. (2009) and Justo-Guillén et al. (2019). Most often, the implemented paradigm is “oddball”—a succession of identical sounds, interspersed with different sounds. This is a situation of abstract succession processing, rather than detection of concrete stimuli. The MMN is an important pointer to the ability of the cognitive system to detect abstraction as a novel stimulus, capturing attention and, possibly, enhancing its memorisation via the episodic memory function to integrate novelty with a context for subsequent, more elaborate processing (Tulving, 1986).
N170—Response to Facial Expressions—The Role of Distinctiveness in Memorisation of Concrete Features
The other important finding of the Dubal et al. study is the elicited later and lower N170 component by the machine face than the human face on the 19 most parietal–occipital electrode sites. It validates the overall experimental paradigm, since N170 reflects the structural encoding in face processing (Carmel and Bentin, 2002). Therefore, although the human viewer is completely aware that the emotion is conveyed by a machine, the goal is achieved in terms of understanding and memorisation of the presented by the teacher knowledge without any internal conflict taking place. In this way a robot can efficiently perform as a skilled assistant to the teacher, well accepted and enjoyed by the learners in this role.
Support to this view is provided by a study of different methodology than the one presented here. The oxygenated (HbO) and deoxygenated (HbR) hemoglobin levels were measured with a continuous wave functional near-infrared spectroscopy (fNIRS) system (measuring two wavelengths 690 and 830 nm) in Jung et al. (2016). This indirect method is almost as popular as EEG and is measuring changes both in oxy- and deoxyhemoglobin concentration but only from regions near the cortical surface. Evoked potentials (EPs) detect the cortical response to a given stimulus with high temporal resolution. fNIRS localises changes in oxygen metabolism following the neural activation. This phenomenon is called neurovascular coupling, allowing parallelism in the interpretation of the observed brain effects by both methodologies (Chiarelli et al., 2017).
In the study by Jung et al. (2016), two groups of participants viewed human vs robotic faces—neurotypical and participants with ASC3. The monitored sites are in the vicinity of T5 in the left hemisphere and T6 in the right hemisphere for a total of 14 channels (within the International 10-20 System as a reference for probe placement). Recordings were averaged within the time frame of 3–10 s after the stimulus onset. The neurotypical viewers showed greater right hemisphere lateralisation than the group with ASC when viewing human faces. However, no difference between the groups was observed in the levels of activation when viewing robotic faces. There was no difference in hemispheric activation, too. The authors interpret the finding of identical brain response to the robotic (but not human) faces in both groups as holding pedagogical potential for children with ASC since the processing of robotic faces is not impaired.
Novelty P3—Levels of Response to Novelty—Perceptual vs Semantic Novelty
Another important aspect of the present analysis—in relation to SED—concerns the definition of the concepts of novelty and surprise. For example, children with ASC dislike novelty, which poses obstacles to the teaching process, especially in inclusive classes (e.g., Vivant et al., 2018). On the other hand, the human cognitive system, in general, is attracted by novelty, and this is an important factor in learning (e.g., Mather, 2013).
In Quent et al. (2021), five types of novelty/surprise signals are outlined—item vs context novelty, item vs context surprise, and complete novelty, following the model of Henson and Gagnepain (2010). A model is being described of the interrelation between distributions of the subjective experience of novelty or surprise in terms of existence of prior memory and the current expectation. Item novelty is defined by encountering an item, not seen before. Context novelty is defined in a similar way. Item surprise is defined by observing a familiar item with novel features, and context surprise—by observing a familiar context with novel features. Complete surprise is when item and context are not seen before. However, the item novelty definition seems rather vague—people do not encounter novel items often—after the age of 3—except in specially designed experimental conditions. On the other hand, the context novelty is experienced mainly with respect to the item—a familiar item can exist in a variety of familiar, yet unsuitable contexts. At the same time, new elements of familiar items or contexts seem the most frequent novelty encounters. Yet, if all are accompanied by the experience of surprise—this would be very resource consuming on the part of the cognitive system, since surprise implies certain emotional engagement.
It was shown in Reichardt et al. (2020) that novelty is often processed without being memorised. In order for a novel item or context to be remembered, some level of experiencing surprise should be present. In such a case, children with learning difficulties may experience additional issues with memorising items that they tend to avoid (as illustrated later in Figure 3).
FIGURE 3
FIGURE 3. Area of the distribution of novelty effects in the brain, possibly attracting curiosity in learning new material of children with ASC.
Our understanding of novelty is more straightforward. We assume novelty a dimension along its intensity—from min to max (Figure 4). We do not distinguish context from item, since these are inter-linked during learning, as Tulving has previously justified in numerous experiments (Tulving, 1983; Tulving, 1986; Tulving, 2002). Low novelty of the item-context configuration is defined by the lack of any noticeable new elements. High novelty is defined by a large proportion of novel elements in less familiar contexts.
FIGURE 4
FIGURE 4. Representation of novelty as a subjective dimension from a minimum (Min) experienced novelty to a maximum (Max).
The subjective experience of novelty is often explored within the “oddball” paradigm (e.g., Levi-Aharoni et al., 2020). The oddball paradigm is an experimental procedure where items (meaningful/meaningless; sensory/verbal; pictorial/spoken; etc.) are presented in a succession, and the EEG activation is observed in the range of 250–425 ms after the stimulus onset. In Barry et al. (2016), the oddball paradigm implements as stimuli nonverbal, auditorily presented sounds, bearing functional, rather than semantic content. The observed late subcomponent, following P3a and P3b, called Novelty P3, is said to be associated with the Orienting Reflex (e.g., MacDonald and Barry, 2020). It was observed at a peak latency of 335 ms. In the subsequent experiment, Novelty P3 was observed at a parietal site called parietal Novelty P3 (nP3p). A frontal Novelty P3 (nP3f) with a peak latency of 379 ms was also observed.
Within the oddball stimuli succession, the emergence of a distinct item invokes the Novelty P3 activation at the later stage of P3. The way the distinct item is being defined is an issue of ongoing debate. One way is to assume that it is a novel item, unprimed before (the most common case) (e.g., Nurdal et al., 2021). Another way is to define the distinct item as (a priory) categorically different from the rest, presented for the first time within the list, for example, a 3-letter syllable in a list of integers (e.g., Hunt, 1995). The better memorisation of the odd item gives support to the forwarded idea of the abstract level of cognitive processing in learning new material being given higher priority than the concrete level.
Studies show that the distinctiveness plays an essential role in the formation of episodic memories, with the substantial involvement of the hippocampus area of the brain (e.g., Hunt, 1995; MacLeod and Kampe, 1996; Dimitrova and Wagatsuma, 2011). However, distinctiveness is processed on different levels of abstraction—surface/perceptual/episodic memory based or deep/abstract/semantic memory based. The early activation components are possibly the familiar ones, whereas the late components represent the elaboration as a cognitive effort to integrate the novel item into contexts that become more and more familiar with the active involvement of the retrieval processes.
The midrange of the distribution of novel elements is what drives curiosity in learning, which is often used as a basis for an intuitive teaching strategy by the educators (Figure 5). In Gruber and Ranganath (2019), it was argued that curiosity enhances the attention focus of the learner, involving increased dopaminergic activity, which can affect hippocampus-dependent consolidation of memory traces.
FIGURE 5
FIGURE 5. Area of the novelty distribution, attracting curiosity in learning new material.
Semantic vs Perceptual Novelty—Cognitive Processing Priority of Abstraction Over Distinctiveness
By definition, the intensity of surprise is higher, than of novelty, due to the fact that in novelty no prior expectation is being formed as in surprise. In relation to surprise, Quent et al. (2021) reported observed involvement of the hippocampus combined with increased motivation and exploration (Kang et al., 2009; Gruber and Ranganath, 2019)—psychological processes being largely involved in classroom learning.
It is important to distinguish semantic from perceptual novelty. A perceptually novel stimulus is never seen before and will be observed as Novelty P3. A semantically novel stimulus, however, will have smaller amplitude on Novelty P3, since it is a novel feature of a familiar context, like in a folk story. Imagine a story with unusual or surprising elements. These often qualify as new items in familiar contexts. An example is a peach floating down the river in the folk story of Japan called Momotaro, or “peach boy”. When Momotaro meets a dog, a bird, etc., these animals produce sounds, unexpected and unfamiliar, which brings an effect on the part of the listener that is perceptual in nature (and their EEG manifestation will be explored in the future).
The intensity of the perceptual component in the experienced surprise is possibly bigger than the intensity of the semantic component. The reason is that no expectation has been formed during listening to the story for the forthcoming sounds, whereas the semantic processing relies on expectations how the story might develop—either correct or incorrect. When the expectation is incorrect—the person experiences some form of surprise (often negative as emotion). It is assumed that the elaboration, associated with processing of the perceptual distinctiveness, is higher than during semantic processing, so more cognitive resource will be involved and the learning will be felt as more effortful.
It was shown in a study implementing eye tracking that children with ASC fail to form correct expectations about the development of a video plot just before the emerging conflict, whereas neurotypical (NT) children successfully form correct anticipations about the video plot. The method of determining this is recording the blink rate, where blink suppression is observed just before the conflict (NT), or a little after it (ASC) (Shultz et al., 2011). Since the intensity of the experienced surprise is higher than that of the novelty, children with ASC are under stress more often, than the NT children. Being prone to sensory-perceptual stimulation, they meet difficulties in forming or retrieving abstractions, needed in order to be at advantage during learning (e.g., Vanpaemel and Bayer, 2021). This leads to the involuntary shift of the focus of curiosity toward the left side of the distribution of the subjective novelty dimension (Figure 3) in order to spare cognitive resource.
Such an interpretation of abstraction processing by children with ASC is in line with the previously forwarded by the authors of the present article “limited resource hypothesis” (Dimitrova et al., 2015). According to it, children with ASC involuntarily shift attention towards the automatic range, that is, are more susceptible to influences, eliciting involuntary attention and curiosity (the left side of the distribution of novelty effects in the brain, depicted in Figure 3). The right side of this distribution involves more elaborate mental processing of voluntary attention, driving curiosity (experienced as more effortful by all children).
The comparison of the striped areas in Figure 5 and Figure 3 in relation to the possible diagnostic ability of the cyber-physical system (CPS) in the present context is hypothetical. It is possible and feasible to further test this hypothesis by designing purely behavioural tests/games to involve children and assess their learning skills. The focus is on curiosity—whether the distribution of the expressed attraction to the task is symmetrical in a group of children—or skewed in one of the directions—the automatic processing range (involuntary attention), left, or the consciously controlled range (voluntary attention), right. The results in any case will provide valuable insights into the underlying neuro-cognitive processing of the human brain without involving any direct EEG measurement method.
Learning Processes Revealed by Observing the EEG Activation Associated With Reward Processing—RewP (250–450 ms)
The reward positivity (RewP) is a positive going fronto-central component, peaking around 250–450 ms after evaluative feedback of subject performance, being larger, when the reward is bigger (Brown et al., 2020). It is associated with mechanisms of the brain, acting as a predictive system, aiming to spare energy by constantly forming anticipations of future sensory stimulation, or events, as surveyed by Reichardt et al. (2020). One can imagine the experience when learning to drive a car—beginners experience surprise much more often than the experienced drivers, who are, nevertheless, often placed in situations of constant processing of novelty. Interestingly, intrinsic motivations (acting as a substitute of the reward in human learning) seem linked to novelty and surprise processing, rather than processing of unexpected or expected reward (Düzel et al., 2010; Caligiore et al., 2015). Moreover, Meng and Ma (2015) call the difference in the activation between success and failure outcomes of feedback-related negativity (250–350 ms) at the midline-frontal electrodes (d-FRN) “a real-time electrophysiological marker of intrinsic motivation (p. 445)”. Pornpattananangkul et al. (2019) for the first time defined EEG markers of motivation to play a game as different from the time for the choice to be made in the game (i.e., level of difficulty of the choice). Moreover, processing motivation preceded processing how easy was to make one choice over the other-around 300–600 ms at central-parietal sites, but motivation influences occurring around 200 ms earlier. This finding is yet another view supporting the multidimensional type of brain processing underlying learning and decision making. A future idea to be explored is the separate link to memorisation of processing novelty on the one hand and processing reward on the other. It is possible that the subjective dimensions of novelty and reward processing are orthogonal to each other as in Figure 6. Figure 6 presents a hypothetical visualisation of the possible intersection of the novelty processing with the reward processing distributions in the subjective cognitive space in the case of, possibly, similar standard deviations (A) or dissimilar standard deviations (B).
FIGURE 6
FIGURE 6. Visualisation of the possible intersection of the novelty processing (white) with the reward processing (striped) distributions in the subjective cognitive space in the case of similar standard deviations (A) and dissimilar standard deviations (B).
The intersection of the probability distributions represent the simultaneous effect of the underlying brain processing of novelty and reward along the theory of Torgerson (1958, p. 247). Therefore, it is promising to model the brain as a multidimensional processing system of various subjective states, like the association of processing reward and memorisation, vs novelty and memorisation, possibly with differential involvement of emotion. These aspects need to be explored in future studies.
It was also shown that processing novelty is related to curiosity as a state and a personality trait (Gruber and Ranganath, 2019). This novel finding demonstrates the feasibility to study brain processes of limited temporal scope (internal states) as separate from long-term, belonging to the personality of the learner. High curiosity as a trait requires a different way to approach the learner by the teacher than low curiosity. Yet accounting for the personality of the child is a factor to improve learning, especially in children with special learning needs.
The personality dimensions, theoretically, are infinite in number (e.g., Dimitrova, 2016). Two novel types of personalities were defined in a human-robot interaction context—socially motivated and socially indifferent (Dimitrova et al., 2019). Interestingly, the socially motivated learners in a zoology lesson, taught by a humanoid robot, gave reliably more positive comments on the robot itself, than the socially indifferent learners (contrary to any ad hoc expectation). This was interpreted as providing support to the socialising role of humanoid robots in class. The relation of social motivation, curiosity and the intrinsic motivations within a multidimensional brain processing system is still to be explored in future studies.
The “Surprising” Role of Dispositions in Semantic Processing, and of Predictive Processing on the Uncanny Valley Subjective State—The Effects on N400
An EEG marker of semantic processes is a negative wave, the so-called N400 in response to words that cannot be integrated semantically into the sentence context, occurring at midline electrode sites (Fz, FCz, Cz, CPz, and Pz) (Dwivedi and Selvanayagam, 2021). Interestingly, people with negative personal disposition elicited attenuated N400. No correlation of the expression of positive disposition and the amplitude of N400 was observed. This finding is essential in focusing on children’s dispositions during learning, especially if the focus is on preserving cognitive resource. Further studies may bring interesting insights on the role of the dispositions on the learning outcome so that psychosocial and pedagogical rehabilitation4 approaches are also included in classroom work to make it more enjoyable in the future.
In Urgen et al. (2018), a robot (mechanical appearance), an android (human-like appearance), and a human performed everyday actions—drinking from a cup, waving hand, etc. The movements of the robot and the human were called congruent, since they were what could be expected—abrupt, robot-like for the robot and smooth, human-like for the human. The android, however, performed incongruent movements—since it moved abruptly (being mechanical), but looking like a human. EEG was recorded from human participants after a still, or a moving video of the stimulus. As expected by the authors, the N400 effect at the frontal sites (AF3, AFz, AF4, Fz, F1, F2, F3, F4, F5, and F6) of the moving android was the strongest in comparison to the rest, as well as when dynamic vs static conditions were compared. The explanation was that viewers formed anticipation of the behaviour of the observed agents—human, robotic, or android. The expected congruence is the strongest factor, which can be used as a guideline in the design of educational robotic systems for pedagogical rehabilitation in inclusive classes.
Processing Abstraction at the Highest Level—Syntax and Semantic Effects on P600 and Detection of Mind Wandering
Syntactic processes are linked to a late centro-parietal positivity, termed P600, which occurs between 600 and 1,000 ms. The P600 correlates with processing of syntactically complex sentences (Daltrozzo and Conway, 2014). Delogu et al. (2019) claim that N400 is related to retrieval, rather than semantic processing, whereas P600 correlates with the difficulty to meaningfully integrate the sentence, especially when the sentence is syntactically correct but semantically improbable. According to Brouwer and Crocker (2017, p. 4) “… N400 amplitude is sensitive to the degree of semantic expectancy, while P600 amplitude is modulated by the nature of the comprehension task. It is the interplay between the systematic modulation of these latent components, due to their spatiotemporal overlap in the observed ERP signal, which explains the variance in WCS derived effects … ”, referring to the waveform component structure (WCF), underlying most of the mainstream analysis and explaining some observed inconsistencies. The abovementioned temporal overlap is consistent with the multidimensional view of brain processing, especially in a complex domain, such as simultaneous semantic and syntactic processing.
Table 1 presents summarisation of the EEG markers of abstraction, attention and memory in the present survey. The final column presents the respective pedagogical implications, substantiated by the discussed brain processes.
TABLE 1
TABLE 1. Summarisation of the EEG markers of abstraction, attention, and memory in the present survey.
In our future studies, we plan to use folk stories told by humanoid robots. The stories typically contain elements, requiring semantic integration, whereas syntactically these are consistent. An example is a boy jumps out of the peach, after it is cut in halves. These studies will enable more detailed investigation of processing of abstract information at the late event-related potential (ERP) activations during comprehension of a story and prediction of its later recall.
One perspective educational approach would be the detection of “mind wandering” during learning, which will help the teacher reduce the number of stressful examinations on the entire study material. Groot et al. (2021) report a study of combined fMRI-EEG and pupilometry on identifying markers of mind wandering in tasks, requiring attention on the part of the learner. It demonstrated the feasibility of 1) combined registration of neurological and behavioural (pupilomery) markers of subjective states and 2) possibility to obtain reliable markers of mind wandering in principle. For example, it was shown that pupil dilation during mind wandering is bigger than during the on-task attention. Attention deficit/hyperactivity disorder (commonly observed in children with ASC) was predicted by a set of EEG markers of spontaneous mind wandering in (Bozhilova et al., 2018). This type of studies is in its early stage, yet holding a large potential for future classroom situations.
Conclusion
The article presented a survey of literature for a developed approach to implementing CPS, capable of diagnosing learner states so that the process of acquiring new knowledge becomes easier, more enjoyable and less effortful in future inclusive classes. The CPS will have to be endowed with an improved diagnostic ability for efficient technological support of the learner according to their individual style, pace, and interests.
In the present article, we focus mainly on EEG markers of involuntary, implicit, and spontaneous processing, since our aim is to identify strategies that capture learner attention and interest and avoid the effortful style of current education frameworks. We believe that the potential of the “amusing education” is extremely essential and capable of benefitting from the numerous present-day cognitive and neuro-cognitive studies.
A second specificity of the approach, presented in the article, is not focusing on the numerous studies, comparing the cognition of learners with ASC and of neurotypical students. The reason is the adopted approach to explore relevant knowledge from the general studies on brain processing in a variety of experimental paradigms. This type of knowledge is still largely unexplored in a systematic manner in relation to design of pedagogical scenarios and can be used to a greater extent in the design of novel pedagogical situations, including humanoid robots as valuable assistants of the teachers in class.
Author Contributions
MD and HW are responsible for the style of the article, the main ideas, and the overall format. AK contributed the figures. AK, EV, and JDN-G contributed to the definition of the conceptual framework.
Funding
The presented research received funding from the EC for project CybSPEED, № 777720, H2020-MSCA-RISE-2017; the Bulgarian Research Fund project, № KP-06-H42 /4 (2020–2023); and the project Competence Center “Intelligent mechatronic, eco- and energy saving systems and technologies” № BG05M2OP001-1.002-0023, OP Science and Education for Smart Growth (2014–2020).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
1Similar to the definition of “family resemblance” of Eleanor Rosch (Rosh and Mervis, 1975).
2The file with the figure is made available under the Creative Commons CC0 1.0 Universal Public Domain Dedication.
3We favour the approach, defining autism as a condition (ASC), but not disorder (ASD) (e.g., Cheng et al., 2010; Dimitrova, 2015).
4https://cordis.europa.eu/project/id/777720
References
Bornkessel-Schlesewsky, I., and Schlesewsky, M. (2008). An Alternative Perspective on “Semantic P600” Effects in Language Comprehension. Brain Res. Rev. 59 (1), 55–73. doi:10.1016/j.brainresrev.2008.05.003
Bozhilova, N. S., Michelini, G., Kuntsi, J., and Asherson, P. (2018). Mind Wandering Perspective on Attention-Deficit/hyperactivity Disorder. Neurosci. Biobehavioral Rev. 92, 464–476. doi:10.1016/j.neubiorev.2018.07.010
Brouwer, H., and Crocker, M. W. (2017). On the Proper Treatment of the N400 and P600 in Language Comprehension. Front. Psychol. 8, 1327. doi:10.3389/fpsyg.2017.01327
Brown, D. R., Richardson, S. P., and Cavanagh, J. F. (2020). An EEG Marker of Reward Processing Is Diminished in Parkinson's Disease. Brain Res. 1727, 146541. doi:10.1016/j.brainres.2019.146541
Caligiore, D., Mustile, M., Cipriani, D., Redgrave, P., Triesch, J., De Marsico, M., et al. (2015). Intrinsic Motivations Drive Learning of Eye Movements: an experiment with Human Adults. PLoS One 10 (3), e0118705. doi:10.1371/journal.pone.0118705
Carmel, D., and Bentin, S. (2002). Domain Specificity versus Expertise: Factors Influencing Distinct Processing of Faces. Cognition 83 (1), 1–29. doi:10.1016/S0010-0277(01)00162-7
Cheng, Y., Chiang, H.-C., Ye, J., and Cheng, L.-h. (2010). Enhancing Empathy Instruction Using a Collaborative Virtual Learning Environment for Children with Autistic Spectrum Conditions. Comput. Edu. 55 (4), 1449–1458. doi:10.1016/j.compedu.2010.06.008
Chiarelli, A. M., Zappasodi, F., Di Pompeo, F., and Merla, A. (2017). Simultaneous Functional Near-Infrared Spectroscopy and Electroencephalography for Monitoring of Human Brain Activity and Oxygenation: a Review. Neurophoton 4 (4), 1041411. doi:10.1117/1.NPh.4.4.041411
Daltrozzo, J., and Conway, C. M. (2014). Neurocognitive Mechanisms of Statistical-Sequential Learning: what Do Event-Related Potentials Tell Us? Front. Hum. Neurosci. 8, 437. doi:10.3389/fnhum.2014.00437
Delogu, F., Brouwer, H., and Crocker, M. W. (2019). Event-related Potentials index Lexical Retrieval (N400) and Integration (P600) during Language Comprehension. Brain Cogn. 135, 103569. doi:10.1016/j.bandc.2019.05.007
Dimitrova, M. (2015). Gestalt Processing in Human-Robot Interaction: a Novel Account for Autism Research. BRAIN. Broad Res. Artif. Intelligence Neurosci. 6 (1-2), 30–42. available at: https://lumenpublishing.com/journals/index.php/brain/article/view/1961.
Dimitrova, M., Krastev, A., Zahariev, R., Vrochidou, E., Bazinas, C., Yaneva, T., et al. (2020). Robotic Technology for Inclusive Education: A Cyber-Physical System Approach to Pedagogical Rehabilitation. CompSysTech '20: Proceedings of the 21st International Conference on Computer Systems and Technologies ’20, 293–299. doi:10.1145/3407982.3408019
Dimitrova, M. (2016). “Towards Design of High-Level Synthetic Sensors for Socially-Competent Computing Systems,” in Revolutionizing Education through Web-Based Instruction Hershey, PA: IGI Global, 20–34. doi:10.4018/978-1-4666-9932-8.ch002
Dimitrova, M., and Wagatsuma, H. (2015). Designing Humanoid Robots with Novel Roles and Social Abilities. Lovotics 3 (1). available at: https://www.omicsonline.org/open-access/designing-humanoid-robots-with-novel-roles-and-social-abilities-2090-9888-1000112.php?aid=67398. doi:10.4172/2090-9888.1000112
Dimitrova, M., Wagatsuma, H., Tripathi, G. N., and Ai, G. (2015). Adaptive and Intuitive Interactions with Socially-Competent Pedagogical Assistant Robots. In 2015 IEEE International Conference on Information Technology Based Higher Education and Training ITHET, 1–6. doi:10.1109/ITHET.2015.7218031
Dimitrova, M., Wagatsuma, H., Tripathi, G. N., and Ai, G. (2019). Learner Attitudes towards Humanoid Robot Tutoring Systems. Cyber-Physical Syst. Soc. Appl. Hershey, PA: IGI Global, 62–85. doi:10.4018/978-1-5225-7879-6.ch004
Dimitrova, M., and Wagatsuma, H. (2011). “Web Agent Design Based on Computational Memory and Brain Research,” in Information Extraction from the Internet, Hong Kong: iConcept Press Ltd., 35–56. doi:10.5281/zenodo.4768125
Dubal, S., Foucher, A., Jouvent, R., and Nadel, J. (2011). Human Brain Spots Emotion in Non Humanoid Robots. Soc. Cogn. Affective Neurosci. 6, 90–97. doi:10.1093/scan/nsq019
Dunbar, R. I. M. (2003). The Social Brain: Mind, Language, and Society in Evolutionary Perspective. Annu. Rev. Anthropol. 32 (1), 163–181. doi:10.1146/annurev.anthro.32.061002.093158
Duncan, C. C., Barry, R. J., Connolly, J. F., Fischer, C., Michie, P. T., Näätänen, R., et al. (2009). Event-related Potentials in Clinical Research: Guidelines for Eliciting, Recording, and Quantifying Mismatch Negativity, P300, and N400. Clin. Neurophysiol. 120 (11), 1883–1908. doi:10.1016/j.clinph.2009.07.045
Düzel, E., Bunzeck, N., Guitart-Masip, M., and Düzel, S. (2010). NOvelty-related Motivation of Anticipation and Exploration by Dopamine (NOMAD): Implications for Healthy Aging. Neurosci. Biobehavioral Rev. 34 (5), 660–669. doi:10.1016/j.neubiorev.2009.08.006
Dwivedi, V. D., and Selvanayagam, J. (2021). Effects of Dispositional Affect on the N400: Language Processing and Socially Situated Context. Front. Psychol. 12, 1045. doi:10.3389/fpsyg.2021.566894
Elsherif, M. M., Saban, M. I., and Rotshtein, P. (2017). The Perceptual Saliency of Fearful Eyes and Smiles: A Signal Detection Study. PloS one 12 (3), e0173199. doi:10.1371/journal.pone.0173199
Groot, J. M., Boayue, N. M., Csifcsák, G., Boekel, W., Huster, R., Forstmann, B. U., et al. (2021). Probing the Neural Signature of Mind Wandering with Simultaneous fMRI-EEG and Pupillometry. NeuroImage 224, 117412. doi:10.1016/j.neuroimage.2020.117412
Gruber, M. J., and Ranganath, C. (2019). How Curiosity Enhances Hippocampus-dependent Memory: The Prediction, Appraisal, Curiosity, and Exploration (PACE) Framework. Trends Cogn. Sci. 23 (12), 1014–1025. doi:10.1016/j.tics.2019.10.003
Henson, R. N., and Gagnepain, P. (2010). Predictive, Interactive Multiple Memory Systems. Hippocampus 20 (11), 1315–1326. doi:10.1002/hipo.20857
Hunt, R. R. (1995). The subtlety of distinctiveness: What von Restorff really did. Psychon. Bull. Rev. 2, 105–112. doi:10.3758/BF03214414
Jung, C. E., Strother, L., Feil-Seifer, D. J., and Hutsler, J. J. (2016). Atypical Asymmetry for Processing Human and Robot Faces in Autism Revealed by fNIRS. PLoS One 11 (7), e0158804. doi:10.1371/journal.pone.0158804
Justo-Guillén, E., Ricardo-Garcell, J., Rodríguez-Camacho, M., Rodríguez-Agudelo, Y., Lelo de Larrea-Mancera, E. S., and Solís-Vivanco, R. (2019). Auditory Mismatch Detection, Distraction, and Attentional Reorientation (MMN-P3a-RON) in Neurological and Psychiatric Disorders: A Review. Int. J. Psychophysiology 146, 85–100. doi:10.1016/j.ijpsycho.2019.09.010
Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure, S. M., Wang, J. T.-y., et al. (2009). The Wick in the Candle of Learning. Psychol. Sci. 20 (8), 963–973. doi:10.1111/j.1467-9280.2009.02402.x
Lee, T. S., Yang, C. F., Romero, R. D., and Mumford, D. (2002). Neural Activity in Early Visual Cortex Reflects Behavioral Experience and Higher-Order Perceptual Saliency. Nat. Neurosci. 5 (6), 589–597. doi:10.1038/nn0602-860
Levi-Aharoni, H., Shriki, O., and Tishby, N. (2020). Surprise Response as a Probe for Compressed Memory States. Plos Comput. Biol. 16 (2), e1007065. doi:10.1371/journal.pcbi.1007065
Lieberman, M. D. (2007). Social Cognitive Neuroscience: a Review of Core Processes. Annu. Rev. Psychol. 58, 259–289. doi:10.1146/annurev.psych.58.110405.085654
Lindskog, M., Nyström, P., and Gredebäck, G. (2021). Can the Brain Build Probability Distributions? Front. Psychol. 12, 596231. doi:10.3389/fpsyg.2021.596231
MacDonald, B., and Barry, R. J. (2020). Integration of Three Investigations of Novelty, Intensity, and Significance in Dishabituation Paradigms: A Study of the Phasic Orienting Reflex. Int. J. Psychophysiology, 147, 113–127. doi:10.1016/j.ijpsycho.2019.11.009
MacDorman, K. (2005).Androids as an Experimental Apparatus: Why Is There an Uncanny valley and Can We Exploit it. In Proc. Toward Social Mechanisms of Android Science-An ICCS/CogSci. Stresa, 106–118. available at: https://www.lri.fr/∼sebag/Slides/Uncanny_ABernard.pdf.
MacLeod, C. M., and Kampe, K. E. (1996). Word Frequency Effects on Recall, Recognition, and Word Fragment Completion Tests. J. Exp. Psychol. Learn. Mem. Cogn. 22 (1), 132–142. doi:10.1037/0278-7393.22.1.132
Mather, E. (2013). Novelty, Attention, and Challenges for Developmental Psychology. Front. Psychol. 4, 491. doi:10.3389/fpsyg.2013.00491
McLeod, S. (2019). Introduction to the Normal Distribution (Bell Curve). available at: https://www.simplypsychology.org/normal-distribution.html (accessed on 07 14, 2021). doi:10.4324/9780429024245
Moore, R. K. (2012). A Bayesian Explanation of the ‘Uncanny Valley’ Effect and Related Psychological Phenomena. Sci. Rep. 2 (1), 864–865. doi:10.1038/srep00864
Nurdal, V., Fairchild, G., and Stothart, G. (2021). The Effect of Repetition Priming on Implicit Recognition Memory as Measured by Fast Periodic Visual Stimulation and EEG. Int. J. Psychophysiology 161, 44–52. doi:10.1016/j.ijpsycho.2021.01.009
Oxley, B. C. (2020). File:International 10-20 System for EEG-MCN.Png. available at: https://commons.wikimedia.org/wiki/File:International_10-20_system_for_EEG-MCN.png#metadata (accessed 07 14, 2021). This file is made available under the Creative Commons CC0 1.0 Universal Public Domain Dedication.
Pornpattananangkul, N., Grogans, S., Yu, R., and Nusslock, R. (2019). Single-trial EEG Dissociates Motivation and Conflict Processes during Decision-Making under Risk. NeuroImage 188, 483–501. doi:10.1016/j.neuroimage.2018.12.029
Quent, J. A., Henson, R. N., and Greve, A. (2021). A Predictive Account of How novelty Influences Declarative Memory. Neurobiol. Learn. Mem. 179, 107382. doi:10.1016/j.nlm.2021.107382
Read, R., and Belpaeme, T. (2016). People Interpret Robotic Non-linguistic Utterances Categorically. Int. J. Soc. Robotics 8, 31–50. doi:10.1007/s12369-015-0304-0
Reichardt, R., Polner, B., and Simor, P. (2020). Novelty Manipulations, Memory Performance, and Predictive Coding: The Role of Unexpectedness. Front. Hum. Neurosci. 14, 152. doi:10.3389/fnhum.2020.00152
Rosch, E., and Mervis, C. (1975). Family Resemblances: Studies in the Internal Structure of Categories. Cogn. Psychol. 7, 573–605. doi:10.1016/s0278-2626(03)00052-6
Rosch, E. (1987). “Wittgenstein and Categorization Research in Cognitive Psychology,” in Meaning and the Growth of Understanding. Wittgenstein's Significance for Developmental Psychology. Editors M. Chapman, and R. Dixon (Hillsdale, NJ: Erlbaum).
Schartner, M., Seth, A., Noirhomme, Q., Boly, M., Bruno, M. A., Laureys, S., et al. (2015). Complexity of Multi-Dimensional Spontaneous EEG Decreases during Propofol Induced General Anaesthesia. PloS one 10 (8), e0133532. available at: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0133532. doi:10.1371/journal.pone.0133532
Schleiger, E., Sheikh, N., Rowland, T., Wong, A., Read, S., and Finnigan, S. (2014). Frontal EEG delta/alpha Ratio and Screening for post-stroke Cognitive Deficits: the Power of Four Electrodes. Int. J. psychophysiology 94 (1), 19–24. doi:10.1016/j.ijpsycho.2014.06.012
Shultz, S., Klin, A., and Jones, W. (2011). Inhibition of Eye Blinking Reveals Subjective Perceptions of Stimulus Salience. Proc. Natl. Acad. Sci. 108 (52), 21270–21275. doi:10.1073/pnas.1109304108
Thurstone, L. L. (1927). A Law of Comparative Judgment. Psychol. Rev. 34, 273–286. available at: https://brocku.ca/MeadProject/Thurstone/Thurstone_1927f.html. doi:10.1037/h0070288
Tsukida, K., and Gupta, M. R. (2011). How to Analyze Paired Comparison Data. WASHINGTON UNIVERSITY SEATTLE, DEPT OF ELECTRICAL ENGINEERING, UWEE Technical Report, Number UWEETR-2011-0004, available at: https://apps.dtic.mil/sti/pdfs/ADA543806.pdf.
Tulving, E. (2002). Episodic Memory: From Mind to Brain. Annu. Rev. Psychol. 53 (1), 1–25. doi:10.1146/annurev.psych.53.100901.135114
Tulving, E. (1986). What Kind of a Hypothesis Is the Distinction between Episodic and Semantic Memory? J. Exp. Psychol. Learn. Mem. Cogn. 12, 307–311. doi:10.1037/0278-7393.12.2.307
Urgen, B. A., Kutas, M., and Saygin, A. P. (2018). Uncanny valley as a Window into Predictive Processing in the Social Brain. Neuropsychologia 114, 181–185. doi:10.1016/j.neuropsychologia.2018.04.027
Vanpaemel, W., and Bayer, J. (2021). Prototype-based Category Learning in Autism: A Review. Neurosci. Biobehavioral Rev. 127, 607–618. doi:10.1016/j.neubiorev.2021.05.016
Keywords: EEG marker, abstraction, novelty, surprise, involuntary attention, memory, curiosity, cyber-physical systems for special education
Citation: Dimitrova M, Wagatsuma H, Krastev A, Vrochidou E and Nunez-Gonzalez JD (2021) A Review of Possible EEG Markers of Abstraction, Attentiveness, and Memorisation in Cyber-Physical Systems for Special Education. Front. Robot. AI 8:715962. doi: 10.3389/frobt.2021.715962
Received: 27 May 2021; Accepted: 22 July 2021;
Published: 31 August 2021.
Edited by:
Noman Naseer, Air University, PakistanReviewed by:
Hendrik Santosa, University of Pittsburgh, United StatesUmar Shahbaz Khan, National University of Sciences and Technology (NUST), Pakistan
Copyright © 2021 Dimitrova, Wagatsuma, Krastev, Vrochidou and Nunez-Gonzalez. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Maya Dimitrova, m.dimitrova@ir.bas.bg
†These authors have contributed equally to this work