The Contribution of Event-Related Potentials to the Understanding of Sign Language Processing and Production in the Brain: Experimental Evidence and Future Directions

Functional neuroimaging allows investigation of the timing properties of the brain mechanisms underlying covert language processing. This paper presents a review of the use of the neuroimaging technique called Event-Related Potentials (ERPs) in sign language (SL) research. In the field of neurolinguistics, ERPs have been widely used in the study of spoken language, but their use in SL is still rare. Studying the neurocognitive aspects of SL could lead to a better understanding of the specific processing of SL in the brain. This review outlines the basic theoretical and methodological principles of ERPs. We focus on three groups of ERPs that are particularly relevant to SL processing and production: ERPs focusing on cognition, ERPs focusing on language, and ERPs focusing on movement aspects. We then summarize within each group some ERPs that we consider could be useful for studying the sequence of cognitive processes underlying SL processing and we discuss the current state of the use of ERPs within SL research. According to our analysis of the field, ERPs focusing on language aspects have been used more than ERPs focusing on cognitive and movement-related aspects to study SL. More variability in the type of SLs used is needed to expand the inferences made so far. For the development of the field, we recommend the more frequent use of videos and SL stimulation at a natural pace in order to understand how SL is processed in daily life. The use of a wider variety of ERPs in the study of SL is also recommended. We conclude that ERPs offer a useful tool to address unanswered questions in the field, especially those that call for measuring the building blocks of SL processing in real time. The study of SL cognitive processing in the brain is still in its infancy. One way of developing the field in the coming years would be the more frequent use of the ERP neuroimaging technique.

Introduction

The cognitive processing of language (both spoken and signed) is suggested to be the outcome of a sequence of rapid cognitive processes executed over time (Jung-Beeman, 2005; Woll, 2010). This sequence of cognitive processes includes, but is not limited to, sensory, grammatical and semantic processing, and memory retrieval. When these processes are complete, the language item being processed is understood and integrated into the overall meaning in the ongoing discourse. Behavioral and cognitive tasks have given valuable information about this processing. However, usually, they provide only a measure of the overt cognitive outcome, without being able to differentiate the building blocks of language processing. Unfortunately, behavioral techniques alone are not yet good enough to detect the complex organization and coordination of linguistic operations at the cognitive level that support language performance.

The complex interconnection and sequencing of linguistic operations during language comprehension and production are performed very rapidly. One way of studying the timing properties of covert language processing is by focusing on their underlying brain mechanisms. This can be achieved with the use of functional neuroimaging techniques such as electroencephalography (EEG) and magnetoencephalography (MEG). EEG measures the electrical activity generated by the synapses (connections between neurons to transmit information). MEG, on the other hand, measures the magnetic fields generated by the electrical activity produced by the synapses. Thus, they are based on similar measures. Even though there are several differences between EEG and MEG (mainly their sensitivity to the orientation of the sources, for a review see Cohen and Halgren, 2003), regarding the temporal aspects of Event-Related Potentials (ERPs), EEG and MEG can be considered equivalent. In this review, we concentrated on EEG because it is considered a more affordable technology and is therefore more widely used than MEG. Because EEG and MEG are based on the electrical activity generated by the synapses, they are very sensitive to the extremely fast changes produced in this signal.

When a person is involved in a specific task (like language comprehension or production), the underlying electrical activity in the brain can be traced with very exact temporal resolution [in the order of milliseconds (ms)] with the use of EEG/MEG. One way of analyzing the ongoing EEG signals is in the time domain, using ERPs. From non-invasive electrodes attached to the scalp, the ongoing EEG electrical activity is pre-processed (including several procedures to reduce the electrical activity from other sources outside the brain) and averaged to extract the peaks, called ERPs [or Event-Related Fields (ERFs) in the case of MEG]. This review seeks to summarize a group of ERPs useful to research oriented to sign languages (SLs) as well as to provide some examples of studies that have been carried out using this neuroimaging technique.

It is believed that these different waves or peaks represent important stages in the sequential cognitive processing involved in the task. Physiologically, the different ERPs represent voltage changes from the summation of the post-synaptic potentials of a large number of neurons, activated (or inhibited) synchronously in response to a stimulus (Fabiani et al., 2007). The time of occurrence of the ERP (latency) gives essential information about the timing and complexity of the underlying cognitive process (measured in ms). Similarly, the size of ERPs (amplitude) is measured in microvolts (μV) which, it is believed, represent the number of neural resources devoted to processing that specific stimulus, or the cognitive process involved. Together, latency and amplitude give important information about how cognitive processing is performed by the brain. The main elements and processes involved in ERP recording are schematically illustrated in Figure 1.

FIGURE 1

Figure 1. Summary of the main elements and processes involved in the ERP recording. (A) Interconnection of the stimulation computer (which delivers the task to the participant), the set of electrodes covering the surface of the scalp (connected to the EEG equipment), and the EEG acquisition computer (connected to the EEG equipment). The main areas of the brain in the left and right hemispheres are also marked. (B) The signal coming from the three elements is averaged and the ERP is obtained as a function of the amplitude (y-axis and measured in μV) and time (x-axis and measured in ms). The vertical line represents the target stimulus onset. The positive and negative peaks or ERPs are marked according to the positive (upward) and negative (downward) position of the amplitude in the diagram.

Another advantage of using EEG/MEG techniques rather than behavioral tests in the study of the cognitive processing of language is that they limit the influence of the subject's state (or subjectivity), which is an essential aspect of conscious human responses. Because EEG/MEG are based on the brain's electrical activity, the participant cannot easily influence them intentionally (although some of the subject's higher-level cognitive stages can influence some ERPs). Since, then, the unwanted subjectivity usually included in behavioral studies can be overcome by the use of ERPs, they allow us to move toward a more objective measure of human behavior.

It is important to note that EEG has high temporal resolution but low spatial resolution. This happens mainly because of the physical properties of the electric signal. Electrical activity is sensitive to fast changes but can be distorted by, in the case of EEG, parts of the brain, bones, membrane or skin lying between the generators and the recording device (electrodes in the scalp). For this reason, the topographic distribution in the scalp (the location of the electrodes where the signal is recorded) is usually provided. Even though both the topography and the neural generators of an ERP can vary depending on the kind of stimulation and paradigm used, the topography of an ERP does not need to reflect the location of its brain generators. Reporting the information about the topographic distribution of ERPs provides useful information such as the number of underlying components, it helps in identifying ERPs, and it allows replicability of the results between laboratories. In this review, for each ERP we provide information about both the typical topographic distribution and the known neural generators. For the same reasons, in this review we purposely highlighted the use of EEG for research questions more related to temporal aspects of SL processing than to spatial or more localizationist questions.

Each ERP is elicited by a typical task or paradigm. For instance, several language-related ERPs in language comprehension are recorded by using a violation paradigm, as one possibility. In this kind of task, one element of the expression is violated depending on the kind of brain response (grammatical, semantic, etc.) that it is intended to measure. In addition, for recording cognitive-related ERPs, variations of the so-called oddball paradigm are among the tasks frequently used. The oddball paradigm involves the repetition of a frequent stimulus including an unexpected and infrequent stimulus that thus generates a cognitive mismatch response. These paradigms, and others, have been widely used in several neuroimaging studies on the cognitive processing of language. Table 1 summarizes the main characteristics of the paradigms mentioned in this review.

TABLE 1

Table 1. Summary of the main characteristics of the paradigms (tasks used to record certain ERPs) mentioned in this review.

Many studies have been conducted with ERPs as brain signatures of processing and production in spoken language (for a review see Beres, 2017). Sign language (SL) and spoken language (SpL) seem to share several general characteristics as regards processing (Neville et al., 1997; Deng et al., 2020; Emmorey et al., 2020). Despite their similarities, some differences between SL and SpL have also been reported (Gutiérrez et al., 2012; Thompson et al., 2020). In order to study specifically SL, we need to better understand how it is processed by the brain. The specific study of the brain bases of SL could help us to disentangle the basic core of language processing from the processes involved in the different modalities used (auditory, visual). Although there is now a growing number of studies focusing on the underlying brain mechanisms involved in SL, still our knowledge in this field is much more limited than that produced by the wide range of studies conducted in SpL. There is also a need for greater diversity in such studies, looking at different kinds of SLs, in order to differentiate what is inherent to SL in general from what could be arising in only one specific SL type.

Given the usefulness of ERPs within linguistics, this review aims to highlight and contribute to their expansion into the study of the cognitive processing of SL (including both comprehension and production) at the level of brain functioning. In particular, this review is intended for linguists and researchers working with SL who want to extend their research to include a cognitive neuroscience approach in a multidisciplinary manner. Based on the perspective that we chose, we will briefly overview some important components of ERPs (and, by implication, their magnetic counterparts as measured by MEG) related to cognition, language, and movement. In each case we will also offer some examples of how they have been used in recent as well as classic studies within the SL literature. To achieve that goal, we made a search in Google Scholar and three scientific databases (Scopus, EBSCOhost and ProQuest) with the string “sign language” AND “ERP.” The search was performed between September 2020 and February 2021, so, only studies published before March 2021 have been included in this review. For each ERP relevant to SL, we chose two representative studies to serve as examples of their usefulness and application within the field so far. The criteria for inclusion of the studies to be presented here were that it was: (1) An original article, (2) using EEG, (3) with aims including SL processing or production in the brain (not enough to study reading in signers or just gestures), and (4) the methodology was well used and simple enough to be explained in a condensed form. (5) The study used and identified specific ERPs (not just effects in defined time-windows), (6) the ERPs studied were within the group of ERPs selected to be introduced in this review, and (7) the study used mainly visual stimulation. Finally, we will discuss methodological and theoretical considerations, as well as suggestions for future directions.

Cognition-Related ERPs

Based on the stimuli used, there are various kinds of ERPs. Cognitive-related ERPs are measured while the participants are solving cognitive tasks. When that is the case, the resultant ERP is a measure of the brain response involved in the processing of the relevant items (or stimuli) of the task. Because of this, cognition-related ERPs are considered mostly stimulus-locked ERPs. Some cognition-related ERPs are going to be further described (see Figure 2A for a schematic representation of the processing stages of cognition-related ERPs).

FIGURE 2

Figure 2. Schematic representation of the processing stages for (A) cognition-related ERPs, (B) language-related ERPs, and (C) movement-related ERPs, over time.

Mismatch Negativity

The Mismatch Negativity (MMN) (Näätänen et al., 1978) is a small negative-going deflection occurring between 150 and 250 ms after an unlikely deviation appears in a sequence of frequently repeating events. The MMN is elicited irrespective of whether the person is paying attention, and without conscious effort, and for that reason it is considered to be an automatic brain response (Näätänen, 2001). The paradigm employed to record it (called a passive oddball) simultaneously includes: (1) a sequence of standard stimuli interrupted by a rare deviant stimulus (to which no answer is required from the participant) and (2), an attention-demanding task to ensure that the MMN is not overlapped by conscious attentional brain activity. The MMN is extracted from the first part of the paradigm as the difference of deviant minus standard brain activity. Functionally, the MMN is considered to be a measure of a mechanism of deviant detection in the brain (Picton et al., 2000). Its brain generators seem to include different regions of the auditory cortex as well as the frontal lobes, with contributions from other brain structures such as the thalamus and hippocampus (Alho, 1995; Astikainen et al., 2000). For that reason, the MMN is usually recorded from electrodes located in frontocentral areas of the scalp.

The MMN was initially considered to be mainly an auditory-related component but some later studies revealed that it could also be obtained from other stimulation modalities such as visual (for a review see Pazo-Alvarez et al., 2003) or somatosensory (Kekoni et al., 1997). The focus here will be on the visual MMN (vMMN) as the one mostly used in SL studies. The vMMN can be used to study pre-attentional automatic sensory discrimination relying on visual information. The topographic distribution of the vMMN does not always match the one described for the auditory MMN. For the vMMN a broad topographical distribution has been reported including the visual cortex in occipital areas (Berti and Schröger, 2001; Czigler et al., 2002), posterior temporal areas (Woods et al., 1992), parietal regions (Cammann, 1990) and frontal regions (Czigler et al., 2002; Wei et al., 2002) regions. The topographic distribution around these areas could depend on the modality and the kind of stimulation used.

The vMMN was used in a study investigating the interplay between language and the perceptual visual system by studying the effect of SL on deaf people's perception of color categories (Xia et al., 2019). An oddball task was used to examine whether the previously known right visual field advantage in SpL also occurs in 14 native users of Chinese SL. Each trial presented two colored squares (green or blue) flanking a fixation point located in the center of the screen. Infrequently, two types of deviant stimuli (squares with a color variation in the same or a different color category) appeared either on the left or the right of the screen. Results showed that the amplitude of the vMMN evoked by the within-color category deviant was significantly smaller than the vMMN evoked by the between-color category deviant when displayed in either the right or the left visual field. These findings allowed the authors to claim that SL influences participants' color perception by using both brain hemispheres, which suggests a language-related effect on perception. The use of the vMMN also made it possible to allocate the time course of the effect in the early, specifically pre-attentive perceptual, processing stages.

Another recent study (Deng et al., 2020) used the vMMN to explore more closely the similarities or differences between languages that use different modalities (such as SpL and SL). Specifically, it examined whether automatic lexical access (previously reported for SpL in hearing speakers) also occurs in lexical sign processing by deaf signers. To answer this question, 22 deaf adult signers and 22 age-matched hearing non-signers were presented with a sequence of four static images representing Hong Kong SL lexical signs (2) and non-signs (2) using a classic oddball paradigm. The two signs represented the meaning of “nurse” and “beverage.” The two non-signs were created as combinations of the signs. One non-sign was generated by combining the handshape of the lexical sign for BEVERAGE and the location of the lexical sign for NURSE. The second non-sign was generated by combining the location of the lexical sign for BEVERAGE and the handshape of the lexical sign for NURSE. Two conditions were used, a lexical sign condition and a non-sign condition. Signs and non-signs were used in reverse order between conditions. In the lexical sign condition, the two non-signs were presented as standards, while the two signs were presented as deviants. In the non-sign condition, the two signs were presented as standards, while the two non-signs were presented as deviants. Unlike the hearing non-signers, deaf signers exhibited an enhanced vMMN to the lexical sign condition (but not to the non-sign condition) at around 230 ms in the parieto-occipital area. These findings were interpreted as indicating that deaf signers implicitly process lexical signs and that neural response differences between deaf signers and hearing non-signers occur at an early stage in sign processing. The authors concluded that the neural underpinnings of SpL and the underlying neural mechanisms for the lexical processing of sign languages are analogous.

As can be inferred from the studies mentioned above, the use of vMMN in the SL context seems to have increased in recent years. As vMMN has been reported to be sensitive to several kinds of violations [motion direction (Pazo-Alvarez et al., 2004), stimulus orientation (Astikainen et al., 2008), the omission of visual stimuli (Czigler et al., 2006), changes in facial expressions (Astikainen and Hietanen, 2009), changes in symmetry (Kecskés-Kovács et al., 2013b), presentation of left vs. right-hand stimuli (Stefanics and Czigler, 2012), or the gender of a face (Kecskés-Kovács et al., 2013a)], we consider that this ERP component still has many potential uses in the SL context. In particular, vMMN could be useful in broadening our understanding of the similarities and differences between signed and spoken modalities, still an issue with multiple open questions, especially from the point of view of brain functioning.

P300

Another ERP involved in the detection of changes in the environment is the P300, which differs from the previous one in that it gives a conscious and attention-driven perspective. The P300 (Sutton et al., 1965) is considered to be an attention marker (Polich and Comerchero, 2003). It is a positive wave with a maximum peak around 300 ms (250–500 ms) after the onset of the variation. It is usually recorded by using an active oddball paradigm in which an unexpected stimulus (target) occurs in a sequence of frequent and repetitive stimuli. The participant is required to respond as fast as possible when the target is detected. Functionally, besides its involvement in attentional processing, it is also considered to be a psychophysiological measure of cognitive function in decision-making processes. Topographically, its maximum peak is usually recorded over the centroparietal areas of the human scalp. The P300 is generated as a result of the interaction between several brain areas, such as the frontal and temporal/parietal networks, with some contributions from several subcortical structures (Ebmeier et al., 1995; Polich, 2003). As the P300 is generated by unlikely targets, the less probable the target, the larger the P300 amplitude (Donchin, 1981). The P300 latency seems to be negatively correlated with mental function, with shorter latencies associated with better cognitive performance (Howard and Polich, 1985; Emmerson et al., 1989).

A recent study (Navarrete et al., 2020) used the P300 to examine the impact of language on sensory visual processing. Based on previous studies suggesting associations between SpL and visual processing, the study explored whether there is an association between SL and the visual recognition of objects based on their orientation. For that purpose, a three-stimuli (drawings of objects) oddball task was presented to 10 deaf native adult signers of Italian SL and 10 hearing non-signer adults. The stimuli appeared in rapid succession, one at a time. The frequent stimuli (with 80% probability of appearance) and infrequent stimuli (with 15% probability of appearance) were the same pictures in different orientations. The target stimulus (with 5% probability of appearance) was another picture with a different orientation from those of the frequent and infrequent stimuli. Participants were asked to respond (by pressing the spacebar of the keyboard) as quickly as possible only when the target appeared on the screen. The analyses focused on the difference between frequent and infrequent trials. Results showed that the P300 elicited by the frequent and infrequent stimuli differed between groups. The P300 amplitude (in centroparietal electrodes) was bigger in signers than in non-signers. The authors conclude that their findings demonstrate that, as in the case of SpL, SL affects the processing of visual objects, thus supporting the idea of an interaction between the linguistic and visual systems, in which the information from both systems integrates with and affects the other (for more information see Ferreira and Tanenhaus, 2007).

Research has identified two subcomponents of the P300: the novelty P300, or P3a, and the classic P300 or P3b. The P3a (Squires et al., 1975) peaks earlier (250 – 280 ms) than the classic P300 and is a positive wave that is maximum over frontocentral electrodes. Functionally, the P3a seems to be associated with the processing of novelty and the orienting subcomponent of attention (related to involuntary shifts of attention to changes in the environment) (Polich, 2003) as well as frontal lobe function (Polich, 2007).

The P3a has been recently used in a study conducted by Ortega et al. (2020) who focused on SL as a second language in an investigation of bilingualism. They tested whether hearing non-signers rely on their gestural repertoire when they are exposed to SL for the first time. Brain electrical activity was recorded when 29 non-signing hearing adults viewed iconic signs that had high and low overlap with the forms of iconic gestures. For example, a sign with high overlap with gestures used in the study is the sign for “to descend” (executed as one hand descending diagonally from right to left) in the Sign Language of the Netherlands. An iconic gesture in Dutch culture to represent “to descend” would also be executed with one hand and a diagonal descending movement. A sign with low overlap with gestures used in this study is the sign for “butterfly” (executed with both hands joined at the base and the fingers opened and moving in and out) in the Sign Language of the Netherlands. An iconic gesture in Dutch culture to represent a butterfly is performed flapping the arms to personify the movement of the insect. Signs with low overlap with gestures elicited a more enhanced positive amplitude in the P3a (with the typical anterior distribution on the scalp) compared to signs with high overlap. The signs with low overlap were interpreted as new to the participants, who therefore allocated more neural resources to processing them. On the other hand, signs with high overlap with gestures could be mapped onto existing gestural schemas. The authors concluded that non-signers generate expectations about the form of new iconic signs based on their implicit knowledge of gestures and not only from their linguistic experience.

Although the P300 is widely used in the field of cognitive neuroscience, its use in the SL research field is, according to our survey, still in its infancy. This ERP component could be particularly useful for isolating the cognitive processes involved in SL processing, such as attention focusing, novelty processing, and decision-making. A specific linguistic domain that could be investigated with P300 is iconicity (of different signs and their parts), for which the study by Ortega et al. (2020) provides an excellent basis. The involvement of cognitive processes could be studied in association with both the SL comprehension and production of children acquiring a SL, adult SL learners, and adult L1 signers.

Language-Related ERPs

Various ERPs have been used to address more specific language-related questions. These ERPs have also used stimulus-locked tasks at different levels, such as vowels/consonants, syllables, words and sentences. In this review, we focus on the brain-based processing of sentence comprehension. Angela Friederici's model (Friederici, 2002) seeks to explain how language comprehension is performed in the brain. Basically, it assumes that sentence comprehension is performed as a succession of at least three main functionally distinct processing stages: an early parsing phase (indexed by the ELAN, see section Early Left Anterior Negativity), followed by processes including semantic information (indexed by the N400, see section N400) and morphosyntactic information (indexed by the LAN, see section Left Anterior Negativity), and a late phase of revision (indexed by the P600, see section P600), in which the semantic and syntactic information are integrated. In the model it is assumed that each stage produces and passes on information to the next stage about the kind of information that has been gathered.

It is worth noticing that some aspects of Friederici's model have been questioned. For instance, it is a syntax-first model stating that the building of syntactic structure precedes semantic processes and that these interact only during a later stage. While some studies have supported the assumption of a relative independence of the semantic and syntactic processing (Gunter et al., 1997; Osterhout and Nicol, 1999), others have found that, when they are combined, semantics and syntax at the brain level seem to interact. Specifically, Hagoort (2003) found that semantics and syntactic brain correlates not only interact, but also affect each other in different ways. Brain-based syntactic analysis seems to be unaffected by semantic integration while semantic integration seems to be harder in the presence of a syntactic problem. These findings are in line with more recent studies (Kemmerer et al., 2007; Kuperberg et al., 2007) describing P600 effects for semantic violations (see subsection P600).

The aforementioned violation paradigm, in which a word is violated (with reference to syntactical or semantical/world knowledge) in relation to the context (the sentence/text), is usually used to record ERPs related to each of the three phases of Friederici's model. It is important to note that there are other paradigms also used in language-related studies, such as “picture naming” (in the case of language production) or “lexical decision” (in the case of language comprehension), that we will not discuss here. The language-related ERPs that we will now go on to discuss can be identified as brain signatures for each of these phases of Friederici's model by the use of the violation paradigm (see Figure 2B for a schematic representation of the processing stages of language-related ERPs). The main methodological differences between the stimulus-locked and the response-locked ERPs are shown in Figure 3.

FIGURE 3

Figure 3. (A) Schematic representation of the methodological difference between the stimulus-locked and the response-locked ERPs. (B) The types of ERP introduced in this review are classified as stimulus-locked or response-locked ERPs.

Syntax-Related ERPs

Early Left Anterior Negativity

The early left anterior negativity (ELAN)¹ (Friederici et al., 1993) is a negative-going wave peaking around 100 to 300 ms after a grammatical violation occurs. As its name indicates, the ELAN is recorded more usually by electrodes located over the left frontal areas of the scalp. This component is recorded in tasks including sentences where the lexico-grammatical category (cf. “word-class”) of words is violated. Interestingly, the sentence does not need to be semantically understandable, thus suggesting that syntax and semantics might be processed independently in the early stages of brain processing. Thus, functionally, the ELAN is considered to reflect early automatic syntax processing in the brain and it is a specific ERP for the evaluation of word category. It has been recorded in both visual and auditory modalities. The ELAN brain generators seem to be located in the temporal as well as in fronto-lateral regions bilaterally, but with dominance in the left hemisphere (Friederici et al., 2000).

One example of the use of the ELAN within the SL context is provided by a study carried out by Capek et al. (2009). This study aimed to determine whether the brain processes indexed by language-related ERPs are specific to aural-oral language or if they underlie any natural language, like in the case of SL. Fifteen deaf native adult users of ASL were presented with sentences signed at a natural rate that were either correct or contained a syntactic error. The syntactic errors were two types of violation in verb modification: reversed verb modification and unspecified verb modification (referred to as verb agreement violations by the authors). In the reversed verb modification condition the verb was moved from the object to the subject, instead of in the opposite direction. In the unspecified verb modification condition the subject-like element and object-like element were set up in space, and instead of directing the verb to either of these locations, the signer directed the verb to a different location in space that had not been previously established. The reversed verb modification condition elicited an ELAN and a later posterior positivity (P600, which will be explained below). The ELAN to the reversed verb modification showed the typical left frontal distribution on the scalp.

Apart from the rather limited use of the ELAN in the spoken/reading language context, according to our survey its use in the study of SL cognitive processing at the brain level is still minimal. This component could be particularly useful in the study of early syntax processing, such as the acceptability of different word-orders², which, to our knowledge, has not yet been addressed in neurolinguistic studies on SLs.

Left Anterior Negativity

Another negativity related to other types of syntax violation is the left anterior negativity (LAN) (Friederici et al., 1996). It is a later ERP than the ELAN, usually recorded between 300 and 500 ms after the grammatical violation occurs in a sentence. It is usually elicited when a syntactically incorrect word is included in a sentence. The kind of task used to elicit the LAN led to the belief that it is related to morphosyntactic processing in the brain during sentence processing. It too is maximally recorded over electrodes located on the left frontal regions of the scalp, but it can also be recorded bilaterally (Hagoort et al., 2003). In Friederici's sentence processing model, the LAN represents the second phase in the processing of morphosyntactic information.

The LAN has been widely employed in language-related studies using spoken and written stimuli. In the last decade, it has started to be used also in the study of brain-based SL processing. For instance, in a study conducted by Hänel-Faulhaber et al. (2014), language-related ERPs were used to determine whether semantic and morphosyntactic aspects of German SL can be dissociated within the same individuals. Fifteen deaf adult native signers watched continuous videotaped German SL sentences (presented sign by sign at natural pace), which were either correct or morphosyntactically incorrect. The morphosyntactic error was performed by a modified verb (incorrect direction of movement) in middle position in the sentences. A LAN was elicited by the sentences with the syntactic violation (referred to as verb agreement violation by the authors) independently of the N400 effect (to be addressed further in the next section) also obtained for the semantic violations. The authors conclude that these results support the idea of a similar neural functional organization of SL and SpL, regardless of the difference in modalities used.

In the previously described study by Capek et al. (2009), the syntactic violation added by the unspecified verb modification elicited a LAN. This component showed a more rightward topographical distribution than the expected leftward maximum. The authors interpreted this hemispheric shift as an effect of the unique involvement of spatial processing in signed syntax, concluding that experience can shape the development of the neural systems that mediate language processing.

Despite the wide use of both ELAN and LAN in spoken language studies, their use in SL processing is still rare, at least according to the survey conducted for this current paper. One reason for this could be the fact that there are still many open questions and even controversies about the syntax of SLs. For example, the distinction between morphology and syntax is not clear, as a single sign can also function as a well-formed grammatical sentence. Studying SL syntax with neurolinguistic methodologies would require that the basic syntactic framework was established. Borrowing conceptual starting points from the SpL research tradition would not serve all our needs. By focusing strictly on SL, ELAN and LAN could help in identifying the nature of the early cognitive mechanisms that underlie syntactic processing in SL. This procedure could help to detect the syntactic features that can be handled at a more basic cognitive level.

Semantic-Related ERP

N400

The most widely used ERP in the language context is the N400 (Kutas and Hillyard, 1980). The N400 is named for its polarity and expected latency (a negative-going deflection peaking around 400 ms after the stimulus onset). The task most typically used to record the N400 (among others using non-verbal stimuli such as figures and faces) involves a sentence (as a context) and a word that is syntactically correct but violates either semantic expectations or world knowledge. This ERP has been recorded in both visual and auditory modalities. Topographically, N400 usually shows a bigger amplitude in centro-parietal areas due to the way the signal is conducted from the sources to the electrodes on the scalp. The neural sources of the N400 include a wide network of brain areas, such as the anterior medial temporal lobe, middle and superior temporal areas, inferior temporal areas, and prefrontal areas (Kutas and Federmeier, 2011). Of these areas, the left hemisphere has been highlighted as an important neural source for the N400, with additional contributions from the right hemisphere (Van Petten and Luka, 2006; Tse et al., 2007). Figure 4 shows the main recording conditions and processes of analysis used to obtain the N400 ERP. With the use of a violation paradigm (panel A), the electrical brain activity is recorded (panel B) over time (x-axis) from a group of electrodes (y-axis) distributed around the scalp. Different marks (triggers) are sent when each kind of target is presented (vertical lines). After averaging the brain response to each kind of target (panel C), the N400 is obtained as the amplitude difference between the brain responses to the neutral targets vs. the brain response to the violated targets (panel D).

FIGURE 4

Figure 4. Schematic representation of the recording and analysis processes involved in obtaining the N400 ERP + example of recording from a signer. (A) Example of a typical paradigm used to elicit the N400 ERP. In each kind of sentence (neutral or violated) the context and the target words are denoted. (B) Plot of raw EEG data as it is recorded over time from different electrodes. The vertical dashed lines represent the onset of each target (critical word) within the sentence. Vertical gray dashed lines show the onset of the target (semantically correct) word for the neutral sentences (NS). Vertical black dashed lines show the onset of the target (semantically violated) word in the violated sentences (VS). (C) Typical averaging of the EEG data after pre-processing (resampling, filtering, etc.) and removing artifacts (e.g., from eye movements, or motor movements). The brain signals corresponding to the onset of the target in each kind of sentence are averaged out to obtain separated brain responses for neutral and violated targets. (D) The resulting evoked potentials locked to the onset of neutral and semantically violated targets from an adult signer when processing actual sign stimulus material in videos. The gray line shows the brain response to the neutral targets, while the black line shows the brain response to the semantically violated targets. The area highlighted in gray denotes the typical time-window for the N400 ERP (between approximately 300–500 ms post target onset). The centro-parietal topography of the N400 ERP is also shown in the Figure.

As the N400 is elicited by semantic deviations in the context of a sentence (although it can also be recorded in word pairs), it is believed to reflect the brain's response to the processing of meaning (Kutas and Hillyard, 1980). Attentional involvement is not necessary to evoke an N400 since the participant is not required to give an answer. The amplitude of the response seems to be proportional to the degree of incongruity of the violation; thus, a more incongruent semantic mismatch would elicit a bigger N400 response than a less incongruent one. The increase in N400 amplitude is functionally interpreted as the processing cost of integrating the meaning of the word into the overall meaning of the sentence.

The N400 is by far the most widely used ERP both in spoken language and in the SL context. In one of the pioneering studies on SL processing with the use of ERPs, Neville et al. (1997) investigated the interplay between semantic processing and the age of acquisition of ASL, including open (noun, verb, and adjectives) and closed (pronouns, conjunctions, and auxiliaries) class elements. Four groups of people were included in the study: two groups of native SL users (one deaf and one hearing group, all with deaf parents), a group of hearing late learners (born to hearing parents, fluent in SL but having learned the language only after reaching 17 years old), and a group of hearing non-signers (with no knowledge of SL and born to hearing parents). The task included sentences in which each sign consisted of eight frames of a digitized film on a screen. Half of the final signs of the sentences were semantically correct given the preceding context and half were semantically anomalous. Each of the signs in the sentences except the first and last was classified as an open or closed-class element. The participants were asked to press one of two buttons to indicate whether or not the sentence made sense. The hearing non-signers were asked to guess whether or not each sentence made sense. Results revealed an N400 that was bigger for open class elements for all the groups. This was interpreted as indicating the existence of substantial similarities in the neural systems mediating the processing of language, independent of the modality through which the language is acquired. Interestingly, the group of hearing late learners showed an increased N400 response to the semantic anomalies, while no clear N400 for semantic violations could be identified in the group of hearing non-signers. The authors claim that these results could indicate that different developmental time courses mediate aspects of semantic processing.

Another study aimed to investigate the influence and time course of lexical access during sentence processing in ASL (Gutiérrez et al., 2012). In this study, the N400 was used to explore the electrophysiological correlates of the processing of handshape (semantic) and location (phonology) during the lexical access of signs. To answer their questions, the authors chose 17 deaf native signers, to whom sequences of pictures representing 100 ASL sentences were presented. The sentences contained 4 types of violation (in the middle of the sentence) regarding the baseline sentences (example: ME FRIEND WEDDING HERS ME BRIDESMAID ME WEAR MUST [target/violated sign: DRESS] EXCITED ME). The 4 types of violation in the sentences included (1) one sign that was semantically congruent in that it did not share location with the baseline (target sign in example sentence: SKIRT) (+S, –P); (2) one sign that was semantically congruent that also shared location with the baseline (target sign in example sentence: SHIRT) (+S, +P); (3) one sign that was semantically incongruent that shared location with the baseline sentence (target sign in example sentence: LUNGS) (–S, +P); and (4) one sign that was unrelated (target sign in example sentence: MIRROR) (–S, –P).

The results obtained by Gutiérrez et al. (2012) showed that the signs that had either only a semantic relationship (+S, –P) or only a phonological overlap (–S, +P) with the expected sign elicited similar N400s, suggesting that semantics and phonology may interact at the level of lexical selection. Interestingly, the N400 elicited by the unrelated (–S, –P) and semantically and phonologically related (+S, +P) conditions exhibited a later onset (450 ms), which was considered to reflect a more difficult semantic integration. All unexpected conditions (+S, –P), (–S, +P), (–S, –P), (+S, +P) exhibited a centrally distributed N400 in the 450–600 ms window, also explained in terms of difficulty in semantic integration with the previous context. It is especially interesting that the N400 elicited by signs sharing only location with the expected sign (–S, +P) showed effects such as stronger and broader distribution, an earlier onset, and also a later offset (persisting into the 600–750 ms window) than the N400 elicited by the other experimental conditions. These results were interpreted as reflecting the special status of location and its potential contribution to semantic interpretation in SL processing, thus replicating at the brain level the results of (Klima and Bellugi, 1979). Based on all these results, the authors concluded that there is an intimate relationship between ASL form and ASL meaning that interacts during online language processing in ways that differ from what has been observed in SpL.

The N400 has been widely used to answer several questions regarding the processing of meaning in the brain in the SL context. It is important to note that there is some controversy over the functional meaning of the N400. Alternative interpretations, based on some interesting experimental findings, consider that the N400 could be reflecting the predictability of the word whose meaning was violated (Nieuwland et al., 2020) or aspects of semantic integration processes (Brown and Hagoort, 1993) rather than the semantic process itself. Experimental manipulation in the SL context could shed some light on this debate.

A Re-analysis or Integration-Related ERP

P600

When a violation paradigm involving grammatical variation is used, the syntactic-related ELAN and LAN are usually followed by a later positivity called the P600, or Syntactic Positive Shift (SPS). This is an ERP elicited by syntactic anomalies of various types (e.g., Osterhout and Holcomb, 1992) and with a maximum peak around 600 ms after the change occurs (onset around 500 ms). It has been recorded in both auditory and visual modalities. Topographically, it is a very widely distributed component spreading over centro-parietal areas of the scalp. Its brain generators are believed to involve Wernicke's area (Service et al., 2007), located in the left temporal lobe. Functionally, it is hypothesized to index the integration of semantic and syntactic information at the neural level. Thus, the P600 seems to index the latest phase in Friederici's sentence processing model.

In the previously presented study by Capek et al. (2009), the biphasic brain response described above to syntactic disagreements formed by ELAN/LAN and P600 components was obtained. Both conditions, the reversed verb agreement and the unspecified verb agreement violations, elicited a P600 with the typical wide medial and centroparietal distribution in the brain. As both P600s showed the typical characteristics of the ERP described for spoken language, the authors claimed that the underlying cognitive operations that give rise to late syntactic processes might be relatively independent of the modality through which the language is perceived.

Also in the study by Hänel-Faulhaber et al. (2014), described above, in which either correct or syntactically modified videotaped German SL sentences were presented to deaf signers, the biphasic pattern of syntactic processing in the brain was present. The modified verbs (incorrect direction of movement) within a sentence elicited, in addition to the LAN, also a late positivity widely distributed in the brain that could be identified with the P600. This component also contributed to differentiating the neural processing of semantic and syntactic aspects as it has been found in oral languages.

Interestingly, several studies have shown the presence of P600 instead of N400 when semantic anomalies representing a violation of a grammatical-semantic constraint are used (e.g., rearranged thematic roles or inverse prenominal adjective order). These results (Kemmerer et al., 2007; Kuperberg et al., 2007) have led to closer examination of the functional significance of the P600 and the language system in general. Kolk and Chwilla (2007) put forward two possible explanations. The first one is consistent with the one presented in this review: that the functional meaning of P600 is related to the integration of semantic and syntactic information. This would support the notion of an interplay between semantics and syntax at the neural level. The second explanation involves the interaction of the language system with the cognitive system. It relates to the involvement of the executive functions in a process of sentence reprocessing promoting of discourse coherence. The executive functions are a group of cognitive processes that are vital for goal-directed behavior such as attention, inhibitory control and working memory (for a review see Diamond, 2013). Thus, the P600 could reflect a reprocessing of sentences to find out whether the cause of a particular inconsistency could be a processing error due to a misunderstanding. Conducting some experiments in which the semantic-syntactic aspects and/or executive cognition are manipulated in SL processing could shed some light on this question. In our opinion, the specificity of SL grammar (e.g., the ability to express information about grammatical relations by modifying the stem of lexemes), provides yet one more channel to explore the two possible explanations for the P600 as a response to semantic violations introduced earlier.

Unfortunately, apart from the studies mentioned above, the P600 has been used hardly at all in the study of the neural correlates of SL processing, according to our survey. This is probably because the nature of SL syntax is still being debated and is not yet fully understood. The combination of P600 with ELAN/LAN could offer a useful tool to distinguish between the early syntactic processing of some SL features and the integration of the syntactic processing into the sentence context, with very exact time tracing. This procedure could help to disentangle the syntactic features that might require a more complex cognitive processing from those that can be handled at a more basic cognitive level. This ERP could even be useful in disentangling core syntax or syntactic phenomena from other types of grammatical (e.g., morphological) phenomena in SLs.

Movement-Related ERPs

Movement-related potentials are ERPs produced before or after a voluntary movement. Thus, they mainly index the neurocognitive processing related to movement preparation and generation. Because they are considered to reflect some aspects of anticipatory behavior when the motor system is involved, these ERPs are classified as response-locked components (see Figure 3). Because SLs are based on the movements (and stopping the movements of) the hands and other parts of the body, we consider that movement-related ERPs are useful research tools to study SL processing. We will now describe some particular movement-related ERPs (see Figure 2C for a schematic representation of the processing stages of movement-related ERPs).

Contingent Negative Variation

One of the ERP components related to preparing for action is the contingent negative variation (CNV). It is a well-known ERP and it is assumed to be a biomarker of expectancy of and preparation for movement in the brain (Walter et al., 1964). It is usually recorded while the participant is solving a task that includes a warning sign (the S1) and another signal (the S2, or imperative stimulus) that indicates a call for a certain kind of response, such as a motor response. The CNV is usually the most prominent component observed in the EEG between S1 and S2. It appears as a negative change of 15–20 μV from the baseline and continues until the S2 onset (Rugg and Coles, 1995). Its maximum peak occurs around 260–470 ms after S1 onset. The topography of the CNV varies with the complexity of the task used. For tasks using mainly motor conditions, the more negative values of the CNV are recorded in the central areas of the scalp (Cui et al., 2000; Brunia and van Boxtel, 2001). Other studies have shown that when the tasks used to elicit the CNV involve more cognitive preparatory processes, it shows a more frontal topographical distribution (Falkenstein et al., 2003). The brain generators of the CNV seem to include multiple areas, including the primary motor cortex, the supplementary motor area, the anterior cingulate cortex, the prefrontal cortex, and the premotor cortex (Hultin et al., 1996; Gómez et al., 2003). Higher motivation for the motor action, a higher requirement of attention, or an S1 that is more prominent or informative could all contribute to increasing the CNV amplitude.

In the language context, the CNV has usually been recorded in rhyme priming tasks. In this kind of task, two items (words, pictures, etc) are sequentially presented to subjects with a long enough interval to separate the neurophysiological and behavioral responses to the first stimulus (S1) and to the second member of the pair (S2, also called the target). Usually, a motor response is needed after the presentation of the S2. The task assumes that exposure to the first (prime) stimulus will influence (by facilitating or inhibiting) the processing of the following stimulus within the pair. The neurophysiological response to S1 usually produces a CNV. An N400 is usually elicited in the neurophysiological response to S2. The combination of CNV and N400 in response to S1 and S2 therefore gives detailed information about the time course of the neural processes involved.

In a study in which they aimed to determine the identity and time course of the neural systems supporting rhyme processing in deaf and hearing adults, MacSweeney et al. (2013) used a rhyme judgment task. In this task, 100 pairs of written rhyming words were presented to nine adult deaf native ASL signers, who showed rhyme judgment performance above chance, and 15 hearing adults. Between the first (S1) and the second (S2) words, a one-second interval made it possible to split the brain responses to the S1 and S2. Results regarding the CNV in response to S1 from the deaf subjects were similar (in polarity, distribution, and timing) to those from the hearing group although, behaviorally, the deaf participants' rhyming abilities were poorer. The authors conclude that the similarity of the CNV modulations between the deaf and hearing groups could indicate sensitivity to the phonological structure of speech in the deaf group, even in the absence of auditory input. They also considered that this result supports the suggestion that phonological processing of written words is to a large degree amodal or supramodal.

The same kind of task and ERP component was used in a similar study aiming to investigate the neural processes underlying both the rhyming and the location judgment abilities of skilled deaf signers as well as of non-signing hearing participants (Colin et al., 2013). Colin et al. (2013) presented 64 pairs of pictures sequentially to 10 deaf native signers and 10 non-signing hearing participants. In the oral language condition (so called by the authors, but not involving spoken language), deaf and hearing participants judged whether the written French name of each picture rhymed or not. In the SL condition, participants judged whether the signs from Belgian French SL corresponding to the two pictures were produced at the same or different locations. A CNV evoked by the first picture of each pair in the oral language condition was present for both groups with a left hemispheric dominance for frontal sites. To the authors, these results suggested that a first phonological encoding of linguistic material is shared by hearing participants (for rhyme) and deaf participants (for rhyme and sign location) hosted in the left hemisphere. The CNV evoked by the first picture of each pair in the SL condition was also present for both groups, but the typical left-hemispheric dominance at the frontal sites was only shown for the group of deaf participants. This result was interpreted as a consequence of early exposure to SL creating a linguistic sensitivity to specific phonological parameters (location). The CNV between both conditions showed the same time course and brain topography for deaf signers, thus confirming previous studies (MacSweeney et al., 2013) claiming that similar neural networks support phonological encoding across modalities.

It should be noted, however, that drawing correspondences between rhyme in SpL and the location of signs in SL is somewhat debatable. For example, the structural parts of signs such as handshape, location and movement may be meaningful, as the use of hand configuration and space in SL may be predominantly motivated rather than arbitrary (e.g., Liddell, 2003; Cormier et al., 2015). It cannot be assumed that structural groupings such as rhymes in SpL and similar sign locations in SL are comparable phenomena, as they may involve different semiotics and patterns of organization.

Although the CNV has mostly been used in SL settings for the study of lexical/phonological processing in signers, it could also be useful for the study of production in SL settings. Since the CNV seems to index the neural processes that occur before movement, it could throw light on the neuroelectric activity related to the preparation of SL production. More specifically, the CNV could help to investigate the more fine-grained cognitive analysis that is performed before a guided movement is executed. As the CNV is usually affected by motivation, attention, and other states of the subject, then those variables could also be manipulated in SL production studies to explore their effects on the neurocognitive correlates of motor preparation before a sign is produced. To the best of our knowledge, no studies tackling this important topic have yet been conducted in the field.

Readiness Potential

Another ERP component indicating the preparation of the brain for an upcoming motor behavior is the Bereitschaftspotential (by its German name) or Readiness Potential (RP, also known as pre-motor potential) (Kornhuber and Deecke, 1965, 2016). It is a measure of brain activity at the cortical level before a voluntary movement is produced, and it is therefore usually recorded when the EEG response is locked to the onset of the movement itself. It is a negative deflection observed from about 1–2 s before the onset of a voluntary action. Topographically, the maximum amplitude of RP is recorded over the motor cortex (left and right precentral and centro-parietal areas of the scalp). The RP's neural generators seem to be located in the primary and supplementary motor areas (Deecke et al., 1982). Other movements, like those of the head, eyes or mouth can affect its recognition and amplitude. The onset of the motor movement needs to be clearly detected to reliably identify the onset of the RP. Thus, usually, the EEG signal is recorded in conjunction with other procedures like Electromyography (EMG) of muscle activity.

The RP has been recorded prior to the motor aspects of speech (Jansen et al., 2014; Wesselmeier and Müller, 2015). Unfortunately, we were unable to find any study conducted in the SL context using the RP. For that reason, here we will describe one study using the RP in the study of some kind of movement production.

In a study based on the assumption that the processing of action verbs and motor planning share common neural substrates, Boulenger et al. (2008) aimed to determine whether the processing of action-related language could interfere with motor action when they are performed concurrently. The authors investigated the impact of a subliminal action word on the neurophysiological correlates of motor preparation (using RP) and on the subsequent execution of an arm-reaching movement (using kinematic analyses). To force the subliminal processing of the words, the words were visually presented, before a signal to execute the arm movement, at a very fast rate, to avoid their being consciously detected and triggering mental motor imagery. Thus, 25 French native speakers were presented with three conditions, one using action words (action verbs), another using non-action words (concrete nouns), and a third, the control (a consonant string), before the signal to move. The results revealed that the slope of the RP that preceded the movement was less negative (indicating a bigger amplitude) following a verb than following a noun. This result suggested that subliminal perception of action verbs could interfere with the concurrent preparation and subsequent execution of a motor movement. The authors concluded that language-related activity in motor areas is indeed part of word processing and that motor areas contribute to the understanding of action words.

Even though the RP seems not to have been a method of choice within SL studies so far, we consider that it would be a useful tool in the study of preparation for action prior to SL production. Motor aspects of language production are usually more studied than the cognitive processes that need to be performed before the movement is initiated. Studying the latter could help us to understand what might be responsible when varying results are achieved in SL production studies. As it has been suggested that the RP reflects planning and anticipation for the forthcoming action (Travers et al., 2021), this ERP offers a good tool to investigate how these cognitive aspects might affect SL production. Methodologically, using SL instead of speech would make the recording of the RP easier, because SL involves a more prominent motor behavior than speech and is therefore easier to identify.

Error-Related Negativity

The error-related negativity (ERN) (Gehring et al., 1990; Falkenstein et al., 1991) or Ne (error negativity) is observed when participants make an incorrect response, even when they are not conscious of it, in simple-choice tasks. It is therefore considered a brain signature of error detection and compensation. The ERN is another component locked to the response, as with other movement-related ERPs. After an error occurs, the ERN appears as a negative-going deflection in the event-related brain potential around 100 ms after the error onset. Topographically, the ERN seems to have a scalp distribution maximal at central and frontal electrode sites (Gehring et al., 1993). It has been suggested that the neural generators of the ERN are located in the dorsal portion of the anterior cingulate cortex (Carter et al., 1998; Miltner et al., 2003) located in the frontal area (surrounding the frontal part of the corpus callosum). Interestingly, the ERN is less evident in correct responses.

In a study examining SL production, Riès et al. (2020) investigated whether the monitoring mechanism of SpL is also involved in SL production. Electrical brain activity was recorded when 20 deaf signers (ASL dominant) and 16 hearing signers (English dominant) were presented with written word-picture pairs (illustrating meaning). Three conditions were included in the task: an identical condition (word and picture coincided semantically), a semantically related condition (word and picture represented different but semantically related objects), and an unrelated condition (word and picture were not related semantically). The participants were instructed to name the pictures as quickly and accurately as possible while ignoring the words. The EEG results revealed an ERN (negativity peaking within 15 ms after the signing onset) in the deaf signers. The ERN was larger in errors than in the correct responses with maximum amplitude over fronto-central scalp electrodes. No clear negativity was present in the hearing signers. Also, the slope of the ERN was correlated with ASL proficiency across signers. From these results, the authors concluded that a similar medial frontal mechanism is engaged in pre-output language monitoring in SL and SpL production.

We were unable to find any report of other research related to the study of SL using an ERN, suggesting a limited usage of this ERP to study SL processing. The ERN (as with the other movement-related ERPs introduced here) could be particularly useful in the study of the cognitive processes preceding SL production. As it is considered to reflect the activation of action monitoring processes in response to erroneous behaviors, its use could be very beneficial in the study of the compensatory processes that are likely to take place during SL production. This top-down cognitive compensatory process could be unequally affected by various factors, including SL proficiency level.

Discussion and Future Directions

ERPs are non-invasive objective correlates of brain activity locked into cognitive processes. They provide basic information about brain cortex activity with exact time-tracing of brain processing, and they can be successfully applied to the study of language processing and specifically to SL studies. As we have shown in the previous sections, ERPs have allowed researchers to significantly expand our knowledge of SL processing and production in the brain. Study of the neurocognitive bases of SL would give access to the undetected cognitive processes that underlie behavior and performance in real-life situations involving SL.

Some ERPs have been used more often than others in SL research, perhaps because of their individual characteristics and how easy they are to record. While language-related ERPs have been relatively popular, movement- and cognitive-related ERPs have been used less. One of the advantages of increasing the number of studies using ERPs in the field would be increased comparability between studies.

Interestingly, the classification of ERPs into stimulus-locked and response-locked components suggests that the two groups of ERPs seem to be used to a greater or lesser extent in the study of different aspects of SL. For instance, stimulus-locked ERPs have mostly been used for examining cognitive and language-related research questions, while response-locked ERPs have been typically used for the study of neural aspects of language (SpL and SL) production. Motor movements are often avoided during EEG and MEG measurements as they can contaminate the signal with unwanted activity from muscles, thus compromising the quality of the results. However, response-locked ERPs (where the brain activity is recorded before the motor movement takes place) offer a useful framework for studying motor-related electrical activity in the brain. When using this methodology, there is even an advantage for SL as compared to SpL. In speech, the articulators are located in the head, close to the brain, making it harder to separate the motor from the brain signals. SL, on the other hand, involves overt movements executed especially by the hands, so the motor response could be more easily detected.

There is still little variability as regards which SLs have been included in studies performed with the use of ERPs. Certain SLs, such as American, British or German SL seem to predominate. This limits the extension of the inferences made by these studies. More studies need to be conducted in other SLs in order to conclude whether the findings already reported for one specific SL are universal, and therefore, applicable to users of other SLs.

Much of the history of modern SL research has been influenced by the view that the structure of SL and the structure of SpL are largely comparable and that the linguistic units and concepts used in SpL research are also applicable in the study of SL. While in some respects this may be true, it should be noted that the different linguistic channels used by SpL and SL are a cause of significant differences between them. For example, signers are able to use two manual articulators (e.g., both hands, as well as the mouth and other parts of the body) (for more information see Puupponen, 2019), which allow meaningful utterances to be constructed in a very different way than those constructed by the speech organs. This may, in fact, also have an effect on the study of the processing of SLs. Problems may emerge when taking concepts from the study of SpL and applying them to the study of aspects of SL that are still the subject of academic debate and that are associated with theoretical claims that still lack empirical support. (Johnston, 2012) 163 points out that the “vast bulk of experiments in the language sciences are conducted using well-described, well-documented languages,” but that “experimental studies of SL users have been conducted just as much to establish the facts of SLs as to test claims about language structure and use.” Although cognitive neuroscience offers new possibilities for tackling relevant issues in the field of SL linguistics, one should exercise caution when considering which claims about SL structure and use are established well enough to be tested, and what we can actually say based on the results. In this respect, studies in closely related fields can shed some light on how SL is processed, and for that reason, they deserve to be included in joint approaches (like in studies comparing SL with gestures).

According to our survey of the field, different types of tasks are used to explore the covert mechanisms involved in several aspects of SL processing. This suggests that by manipulating some SL aspects, their brain neural correlates can be revealed when using neuroimaging techniques. By developing imaginative new tasks to study the linguistic and cognitive properties inherent in SL, several topics that remain understudied in the field could be addressed. For instance, so far, most SL-related ERP studies have avoided the use of videos, perhaps for methodological reasons (as well as limitations of the ERP technique). Because SL is based on movement in space, using video stimulation seems to us to be crucial. This problem could be solved by using clever task manipulations as well as by introducing some modifications into the processes of analysis. With this kind of manipulation, future SL-related ERP research could establish a bridge between laboratory science and the “real-world” settings in which SL interaction occurs. That is, more videos and SL stimulation at a natural pace could be introduced more often to give us more reliable understanding of how SL is processed and produced in daily life. Following this line of thinking, Hosemann et al. (2013) performed an N400 study using videos of naturally signed German SL sentences. Among more specific findings related to their research questions, the authors showed an N400 onset before the critical sign onset (stroke, in terms of Kita et al., 1998). This result suggests that signers also make use of the properties of the transition (or preparation/recovery; see Kita et al., 1998) phase between signs. According to this result, SL comprehension does not seem to start with the beginning of the sign (stroke), but it relies also on valuable information coming from the transition (or preparation/recovery) between one sign (stroke) and the next one. Without using video-stimulation and the ERP technique, it would not be possible to identify this particular attribute of SL comprehension.

Even though ERPs offer multiple advantages, they also have several important limitations that should be considered carefully when applying this methodology to SL research. First, EEG has a low spatial resolution for localizing the neural generators of a particular effect. Although there are some methods (based on mathematical solutions to the inverse problem) for localizing the neural generators of ERPs based on EEG activity, it would be wiser, when answering research questions about localization, to use other techniques with higher spatial resolution, such as MEG or fMRI/MRI. Fortunately, several ERP components have been quite well-studied and their neural generators are well-known. Second, ERPs rely on averaging the stimulus- or response-locked EEG data, on the assumption that the only relevant activity is the one that is kept constant. This procedure might lead to the discarding of important information that is not regular across trials. One way of dealing with this limitation is by analyzing the EEG signal in terms of brain oscillations. Another consideration worth mentioning related to the averaging of EEG activity is that to reliably identify some ERPs, a large number of trials are needed, which will mean extending the measuring times. This requirement could also affect the measurement of rapidly habituating cognitive and neural processes. In that case, the use of trial-by-trial analyses should be considered. As has been mentioned above, it is still difficult to separate brain signals from motion artifacts. This can be considered another limitation of the EEG in general because minimal movements are required from the participants to record reliable data.

In conclusion, ERPs can be considered a very useful tool for understanding different aspects of SL processing that remain uncertain or elusive with the use of other techniques. ERPs could particularly help researchers interested in SL to target, for example, the underlying mechanisms of SL comprehension and production that occur in real time. ERPs' high temporal resolution would allow researchers to determine the sequence of cognitive processes underlying SL processing under a wide variety of task conditions. We believe that they will be used much more widely in SL research in the future.

Author Contributions

DH, AP, and TJ: conception and design of the work, drafting of the article, and critical revision of the article. All authors contributed to the article and approved the submitted version.

Funding

This review has been supported financially by the Academy of Finland under grant 339268.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors wish to thank Eleanor Underwood for the English Language checking of this manuscript.

Footnotes

1. ^In SL research, the abbreviation ELAN typically refers to a software used in video data annotation: the “Eudico Linguistic Annotator,” developed in the Max Planck Institute in Nijmegen, The Netherlands. Obviously, the use of the abbreviation ELAN in this paper has nothing to do with the software.

2. ^Some research claims that the word order of SLs is fixed and stable. However, other research, using different data, claims that the word order exhibits variation. Our motive for suggesting SL word order as a possible target of an ELAN ERP study stems from this conflict. For an overview of studies on SL word order, see Napoli and Sutton-Spence (2014).

References

Alho, K. (1995). Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear Hear. 16, 38–51. doi: 10.1097/00003446-199502000-00004

PubMed Abstract | CrossRef Full Text | Google Scholar

Astikainen, P., and Hietanen, J. K. (2009). Event-related potentials to task-irrelevant changes in facial expressions. Behav. Brain Funct. 5, 30. doi: 10.1186/1744-9081-5-30

PubMed Abstract | CrossRef Full Text | Google Scholar

Astikainen, P., Lillstrang, E., and Ruusuvirta, T. (2008). Visual mismatch negativity for changes in orientation–a sensory memory-dependent response. Eur. J. Neurosci. 28, 2319–2324. doi: 10.1111/j.1460-9568.2008.06510.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Astikainen, P., Ruusuvirta, T., and Korhonen, T. (2000). Cortical and subcortical visual event-related potentials to oddball stimuli in rabbits. Neuroreport 11, 1515–1517. doi: 10.1097/00001756-200005150-00029

PubMed Abstract | CrossRef Full Text | Google Scholar

Beres, A. M. (2017). Time is of the essence: a review of electroencephalography (EEG) and event-related brain potentials (ERPs) in language research. Appl. Psychophysiol. Biofeedback 42, 247–255. doi: 10.1007/s10484-017-9371-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Berti, S., and Schröger, E. (2001). A comparison of auditory and visual distraction effects: behavioral and event-related indices. Brain Res. Cogn. Brain Res. 10, 265–273. doi: 10.1016/S0926-6410(00)00044-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Boulenger, V., Silber, B. Y., Roy, A. C., Paulignan, Y., Jeannerod, M., and Nazir, T. A. (2008). Subliminal display of action words interferes with motor planning: a combined EEG and kinematic study. J. Physiol. Paris 102, 130–136. doi: 10.1016/j.jphysparis.2008.03.015

PubMed Abstract | CrossRef Full Text | Google Scholar

Brown, C., and Hagoort, P. (1993). The processing nature of the N400: evidence from masked priming. J. Cogn. Neurosci. 5, 34–44. doi: 10.1162/jocn.1993.5.1.34

PubMed Abstract | CrossRef Full Text | Google Scholar

Brunia, C. H. M., and van Boxtel, G. J. M. (2001). Wait and see. Int. J. Psychophysiol. 43, 59–75. doi: 10.1016/S0167-8760(01)00179-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Cammann, R. (1990). Is there a mismatch negativity (MMN) in visual modality? Behav. Brain Sci. 13, 234–235. doi: 10.1017/S0140525X00078420

CrossRef Full Text | Google Scholar

Capek, C. M., Grossi, G., Newman, A. J., McBurney, S. L., Corina, D., Roeder, B., et al. (2009). Brain systems mediating semantic and syntactic processing in deaf native signers: biological invariance and modality specificity. PNAS 106, 8784–8789. doi: 10.1073/pnas.0809609106

PubMed Abstract | CrossRef Full Text | Google Scholar

Carter, C. S., Braver, T. S., Barch, D. M., Botvinick, M. M., Noll, D., and Cohen, J. D. (1998). Anterior cingulated cortex, error detection, and the online monitoring of performance. Science 280, 747–749. doi: 10.1126/science.280.5364.747

PubMed Abstract | CrossRef Full Text | Google Scholar

Cohen, D., and Halgren, E. (2003). Magnetoencephalography (neuromagnetism). Encyclopedia of Neurosci. 3, 1–7. Available online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2016.01413/full

Google Scholar

Colin, C., Zuinen, T., Bayard, C., and Leybaert, J. (2013). Phonological processing of rhyme in spoken language and location in sign language by deaf and hearing participants: a neurophysiological study. Neurophysiol. Clin. 43, 151–160. doi: 10.1016/j.neucli.2013.03.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Cormier, K., Fenlon, J., and Schembri, A. (2015). Indicating verbs in British Sign Language favour use of motivated space. Open Linguist. 1, 684–707. doi: 10.1515/opli-2015-0025

CrossRef Full Text | Google Scholar

Cui, R. Q., Egkher, A., Huter, D., Lang, W., Lindinger, G., and Deecke, L. (2000). High resolution spatiotemporal analysis of the contingent negative variation in simple or complex motor tasks and a non-motor task. Clin. Neurophysiol. 111, 1847–1859. doi: 10.1016/S1388-2457(00)00388-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Czigler, I., Balázs, L., and Winkler, I. (2002). Memory-based detection of task-irrelevant visual changes. Psychophysiology 39, 869–873. doi: 10.1017/S0048577202020218

PubMed Abstract | CrossRef Full Text | Google Scholar

Czigler, I., Winkler, I., Pató, L., Várnagy, A., Weisz, J., and Balázs, L. (2006). Visual temporal window of integration as revealed by the visual mismatch negativity event-related potential to stimulus omissions. Brain Res. 1104, 129–140. doi: 10.1016/j.brainres.2006.05.034

PubMed Abstract | CrossRef Full Text | Google Scholar

Deecke, L., Weinberg, H., and Brickett, P. (1982). Magnetic fields of the human brain accompanying voluntary movement. Bereitschaftsmagnetfeld. Exp. Brain Res. 48, 144–148. doi: 10.1007/bf00239582

PubMed Abstract | CrossRef Full Text | Google Scholar

Deng, Q., Gu, F., and Tong, S. X. (2020). Lexical processing in sign language: a visual mismatch negativity study. Neuropsychologia 148, 107629. doi: 10.1016/j.neuropsychologia.2020.107629

PubMed Abstract | CrossRef Full Text | Google Scholar

Diamond, A. (2013). Executive functions. Annu. Rev. Psychol. 64, 135–168. doi: 10.1146/annurev-psych-113011-143750

PubMed Abstract | CrossRef Full Text | Google Scholar

Donchin, E. (1981). Presidential address, 1980: surprise!...surprise? Psychophysiology 18, 493–513. doi: 10.1111/j.1469-8986.1981.tb01815.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Ebmeier, K. P., Steele, J. D., MacKenzie, D. M., O'Carroll, R. E., Kydd, R. R., Glabus, M. F., et al. (1995). Cognitive brain potentials and regional cerebral blood flow equivalents during two-and three-sound auditory “oddball tasks”. Electroencephalogr. Clin. Neurophysiol. 95, 434–443. doi: 10.1016/0013-4694(95)00173-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Emmerson, R. Y., Dustman, R. E., Shearer, D. E., and Turner, C. W. (1989). P3 latency and symbol digit performance correlations in aging. Exp. Aging Res. 15, 151–159. doi: 10.1080/03610738908259769

PubMed Abstract | CrossRef Full Text | Google Scholar

Emmorey, K., Winsler, K., Midgley, K. J., Grainger, J., and Holcomb, P. J. (2020). Neurophysiological correlates of frequency, concreteness, and iconicity in American Sign Language. Neurobiol. Lang. 1, 249–267. doi: 10.1162/nol_a_00012

PubMed Abstract | CrossRef Full Text | Google Scholar

Fabiani, M., Gratton, G., and Federmeier, K. D. (2007). “Event-related brain potentials: Methods, theory, and applications,” in Handbook of Psychophysiology, eds. J. T. Cacioppo, L. G. Tassinary, and G. G. Berntson (Cambridge University Press), 85–119.

PubMed Abstract | Google Scholar

Falkenstein, M., Hohnsbein, J., Hoormann, J., and Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalogr. Clin. Neurophysiol. 78, 447–455. doi: 10.1016/0013-4694(91)90062-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Falkenstein, M., Hoormann, J., Hohnsbein, J., and Kleinsorge, T. (2003). Short-term mobilization of processing resources is revealed in the event-related potential. Psychophysiology 40, 914–923. doi: 10.1111/1469-8986.00109

PubMed Abstract | CrossRef Full Text | Google Scholar

Ferreira, F., and Tanenhaus, M. K. (2007). Introduction to the special issue on language–vision interactions. J. Mem. Lang. 57, 455–459. doi: 10.1016/j.jml.2007.08.002

CrossRef Full Text | Google Scholar

Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends Cognit. Sci. 6, 78–84. doi: 10.1016/S1364-6613(00)01839-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Friederici, A. D., Hahne, A., and Mecklinger, A. (1996). Temporal structure of syntactic parsing: early and late event-related brain potential effects. J. Exp. Psychol. Learn. Mem. Cogn. 22, 1219. doi: 10.1037/0278-7393.22.5.1219

PubMed Abstract | CrossRef Full Text | Google Scholar

Friederici, A. D., Pfeifer, E., and Hahne, A. (1993). Event-related brain potentials during natural speech processing: Effects of semantic, morphological and syntactic violations. Cogn. Brain Res. 1, 183–192. doi: 10.1016/0926-6410(93)90026-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Friederici, A. D., Wang, Y., Herrmann, C. S., Maess, B., and Oertel, U. (2000). Localization of early syntactic processes in frontal and temporal cortical areas: a magnetoencephalographic study. Hum. Brain Mapp. 11, 1–11. doi: 10.1002/1097-0193(200009)11:1<1::AID-HBM10>3.0.CO;2-B

PubMed Abstract | CrossRef Full Text | Google Scholar

Gehring, W. J., Coles, M. G. H., Meyer, D. E., and Donchin, E. (1990). The error-related negativity: an event-related brain potential accompanying errors. Psychophysiology. 27, S34.

PubMed Abstract

Gehring, W. J., Goss, B., Coles, M. G., Meyer, D. E., and Donchin, E. (1993). A neural system for error detection and compensation. Psychol. Sci. 4, 385–390. doi: 10.1111/j.1467-9280.1993.tb00586.x

CrossRef Full Text | Google Scholar

Gómez, C. M., Marco, J., and Grau, C. (2003). Preparatory visuo-motor cortical network of the contingent negative variation estimated by current density. Neuroimage 20, 216–224. doi: 10.1016/S1053-8119(03)00295-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Gunter, T. C., Stowe, L. A., and Mulder, G. (1997). When syntax meets semantics. Psychophysiology 34, 660–676. doi: 10.1111/j.1469-8986.1997.tb02142.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Gutiérrez, E., Williams, D., Grosvald, M., and Corina, D. (2012). Lexical access in American Sign Language: an ERP investigation of effects of semantics and phonology. Brain Res. 1468, 63–83. doi: 10.1016/j.brainres.2012.04.029

PubMed Abstract | CrossRef Full Text | Google Scholar

Hagoort, P. (2003). The interplay between syntax and semantics during sentence comprehension: ERP effects of combining syntactic and semantic violations. J. Cognit. Neurosci. 15, 883–899. doi: 10.1162/089892903322370807

PubMed Abstract | CrossRef Full Text | Google Scholar

Hagoort, P., Wassenaar, M., and Brown, C. M. (2003). Syntax-related ERP-effects in Dutch. Cogn. Brain Res. 16, 38–50. doi: 10.1016/S0926-6410(02)00208-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Hänel-Faulhaber, B., Skotara, N., Kügow, M., Salden, U., Bottari, D., and Röder, B. (2014). ERP correlates of German Sign Language processing in deaf native signers. BMC Neurosci. 15, 1–11. doi: 10.1186/1471-2202-15-62

PubMed Abstract | CrossRef Full Text | Google Scholar

Hosemann, J., Herrmann, A., Steinbach, M., Bornkessel-Schlesewsky, I., and Schlesewsky, M. (2013). Lexical prediction via forward models: N400 evidence from German Sign Language. Neuropsychologia 51, 2224–2237. doi: 10.1016/j.neuropsychologia.2013.07.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Howard, L., and Polich, J. (1985). P300 latency and memory span development. Dev. Psychol. 21, 283. doi: 10.1037/0012-1649.21.2.283

CrossRef Full Text | Google Scholar

Hultin, L., Rossini, P., Romani, G. L., Högstedt, P., Tecchio, F., and Pizzella, V. (1996). Neuromagnetic localization of the late component of the contingent negative variation. Electroencephalogr. Clin. Neurophysiol. 98, 435–448. doi: 10.1016/0013-4694(96)95507-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Jansen, S., Wesselmeier, H., De Ruiter, J. P., and Mueller, H. M. (2014). Using the readiness potential of button-press and verbal response within spoken language processing. J. Neurosci. Methods 232, 24–29. doi: 10.1016/j.jneumeth.2014.04.030

PubMed Abstract | CrossRef Full Text | Google Scholar

Johnston, T. (2012). Lexical frequency in sign languages. J. Deaf Stud. Deaf Educ. 17, 163–193. doi: 10.1093/deafed/enr036

PubMed Abstract | CrossRef Full Text | Google Scholar

Jung-Beeman, M. (2005). Bilateral brain processes for comprehending natural language. Trends Cogn. SciTrends Cogn. Sci. 9, 512–518. doi: 10.1016/j.tics.2005.09.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Kecskés-Kovács, K., Sulykos, I., and Czigler, I. (2013a). Is it a face of a woman or a man? Visual mismatch negativity is sensitive to gender category. Front. Hum. Neurosci. 7, 532. doi: 10.3389/fnhum.2013.00532

PubMed Abstract | CrossRef Full Text | Google Scholar

Kecskés-Kovács, K., Sulykos, I., and Czigler, I. (2013b). Visual mismatch negativity is sensitive to symmetry as a perceptual category. Eur. J. Neurosci. 37, 662–667. doi: 10.1111/ejn.12061

PubMed Abstract | CrossRef Full Text | Google Scholar

Kekoni, J., Hämäläinen, H., Saarinen, M., Gröhn, J., Reinikainen, K., Lehtokoski, A., et al. (1997). Rate effect and mismatch responses in the somatosensory system: ERP-recordings in humans. Biol. PsycholBiol. Psychol. 46, 125–142. doi: 10.1016/S0301-0511(97)05249-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Kemmerer, D., Weber-Fox, C., Price, K., Zdanczyk, C., and Way, H. (2007). Big brown dog or brown big dog? An electrophysiological study of semantic constraints on prenominal adjective order. Brain Lang. 100, 238–256. doi: 10.1016/j.bandl.2005.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Kita, S., van Gijn, I., and van der Hulst, H. (1998). “Movement phases in signs and co-speech gestures and their transcription by human coders,” in Gesture and Sign Language in Human-Computer Interaction: Proceedings of International Gesture Workshop, eds I. Wachsmuth and M. Froelich (Berlin: Springer), 23–35.

Google Scholar

Klima, E. S., and Bellugi, U. (1979). The Signs of Language. Cambridge, MA: Harvard University Press.

Google Scholar

Kolk, H., and Chwilla, D. (2007). Late positivities in unusual situations. Brain Lang. 100, 257–261. doi: 10.1016/j.bandl.2006.07.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Kornhuber, H. H., and Deecke, L. (1965). Hirnpotentialänderungen bei Willkürbewegungen und passiven Bewegungen des Menschen: Bereitschaftspotential und reafferente Potentiale. Pflügers Arch. 284, 1–17. doi: 10.1007/BF00412364

PubMed Abstract | CrossRef Full Text | Google Scholar

Kornhuber, H. H., and Deecke, L. (2016). Brain potential changes in voluntary and passive movements in humans: readiness potential and reafferent potentials. Pflügers Arch. 468, 1115–1124. doi: 10.1007/s00424-016-1852-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Kuperberg, G. R., Kreher, D. A., Sitnikova, T., Caplan, D. N., and Holcomb, P. J. (2007). The role of animacy and thematic relationships in processing active English sentences: evidence from event-related potentials. Brain Lang. 100, 223–237. doi: 10.1016/j.bandl.2005.12.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Kutas, M., and Federmeier, K. D. (2011). Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Annu. Rev. Psychol. 62, 621–647. doi: 10.1146/annurev.psych.093008.131123

PubMed Abstract | CrossRef Full Text | Google Scholar

Kutas, M., and Hillyard, S. A. (1980). Reading senseless sentences: brain potentials reflect semantic incongruity. Science 207, 203–205. doi: 10.1126/science.7350657

PubMed Abstract | CrossRef Full Text | Google Scholar

Liddell, S. K. (2003). Grammar, Gesture, and Meaning in American Sign Language. Cambridge: Cambridge University Press.

Google Scholar

MacSweeney, M., Goswami, U., and Neville, H. (2013). The neurobiology of rhyme judgment by deaf and hearing adults: an ERP study. J. Cogn. Neurosci. 25, 1037–1048. doi: 10.1162/jocn_a_00373

PubMed Abstract | CrossRef Full Text | Google Scholar

Miltner, W. H. R., Lemke, U., Weiss, T., Holroyd, C., Scheffers, M. K., and Coles, M. G. H. (2003). Implementation of error-processing in the human anterior cingulated cortex: a source analysis of the magnetic equivalent of the error-related negativity. Biol. Psychol. 64, 157–166. doi: 10.1016/S0301-0511(03)00107-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Näätänen, R. (2001). The perception of speech sounds by the human brain as reflected by the mismatch negativity (MMN) and its magnetic equivalent (MMNm). Psychophysiology 38, 1–21. doi: 10.1111/1469-8986.3810001

PubMed Abstract | CrossRef Full Text | Google Scholar

Näätänen, R., Gaillard, A. W. K., and Mäntysalo, S. (1978). Early selective-attention effect on evoked potential reinterpreted. Acta Psychol. 42, 313–329. doi: 10.1016/0001-6918(78)90006-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Napoli, D. J., and Sutton-Spence, R. (2014). Order of the major constituents in sign languages: implications for all language. Front. Psychol. 5, 376. doi: 10.3389/fpsyg.2014.00376

PubMed Abstract | CrossRef Full Text | Google Scholar

Navarrete, E., Miozzo, M., and Peressotti, F. (2020). Language can shape the perception of oriented objects. Sci. Rep. 10, 1–9. doi: 10.1038/s41598-020-65455-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Neville, H. J., Coffey, S. A., Lawson, D. S., Fischer, A., Emmorey, K., and Bellugi, U. (1997). Neural systems mediating American Sign Language: Effects of sensory experience and age of acquisition. Brain Lang. 57, 285–308. doi: 10.1006/brln.1997.1739

PubMed Abstract | CrossRef Full Text | Google Scholar

Nieuwland, M. S., Barr, D. J., Bartolozzi, F., Busch-Moreno, S., Darley, E., Donaldson, D. I., et al. (2020). Dissociable effects of prediction and integration during language comprehension: Evidence from a large-scale study using brain potentials. Philos. Trans. R. Soc. B. 375, 20180522. doi: 10.1098/rstb.2018.0522

PubMed Abstract | CrossRef Full Text | Google Scholar

Ortega, G., and Özyürek, A., and Peeters, D. (2020). Iconic gestures serve as manual cognates in hearing second language learners of a sign language: An ERP study. J. Exp. Psychol. Learn. Mem. Cogn. 46, 403. doi: 10.1037/xlm0000729

PubMed Abstract | CrossRef Full Text | Google Scholar

Osterhout, L., and Holcomb, P. J. (1992). Event-related brain potentials elicited by syntactic anomaly. J. Mem. Lang. 31, 785–806. doi: 10.1016/0749-596X(92)90039-Z

CrossRef Full Text | Google Scholar

Osterhout, L., and Nicol, J. (1999). On the distinctiveness, independence, and time course of the brain responses to syntactic and semantic anomalies. Lang. Cognit. Process. 14, 283–317. doi: 10.1080/016909699386310

CrossRef Full Text | Google Scholar

Pazo-Alvarez, P., Amenedo, E., and Cadaveira, F. (2004). Automatic detection of motion direction changes in the human brain. Eur. J. Neurosci. 19, 1978–1986. doi: 10.1111/j.1460-9568.2004.03273.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Pazo-Alvarez, P., Cadaveira, F., and Amenedo, E. (2003). MMN in the visual modality: a review. Biol. Psychol. 63, 199–236. doi: 10.1016/S0301-0511(03)00049-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Picton, T. W., Alain, C., Otten, L., Ritter, W., and Achim, A. (2000). Mismatch negativity: different water in the same river. Audiol. Neurotol. 5, 111–139. doi: 10.1159/000013875

PubMed Abstract | CrossRef Full Text | Google Scholar

Polich, J. (2003). “Overview of P3a and P3b,” in Detection of Change: Event-Related Potential and fMRI Findings, ed J. Polich (Boston: Kluwer Academic Press), 83–98.

Google Scholar

Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol. 118, 2128–2148. doi: 10.1016/j.clinph.2007.04.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Polich, J., and Comerchero, M. D. (2003). P3a from visual stimuli: typicality, task, and topography. Brain Topogr. 15, 141–152. doi: 10.1023/A:1022637732495

PubMed Abstract | CrossRef Full Text | Google Scholar

Puupponen, A. (2019). Towards understanding nonmanuality: a semiotic treatment of signers' head movements. Glossa 4, 39. doi: 10.5334/gjgl.709

CrossRef Full Text | Google Scholar

Riès, S. K., Nadalet, L., Mickelsen, S., Mott, M., Midgley, K. J., Holcomb, P. J., et al. (2020). Pre-output language monitoring in sign production. J. Cog. Neurosci. 32, 1079–1091. doi: 10.1162/jocn_a_01542

PubMed Abstract | CrossRef Full Text | Google Scholar

Rugg, M. D., and Coles, M. G. (1995). Electrophysiology of Mind: Event-Related Brain Potentials and Cognition. Oxford: Oxford University Press.

Google Scholar

Service, E., Helenius, P., Maury, S., and Salmelin, R. (2007). Localization of syntactic and semantic brain responses using magnetoencephalography. J. Cog. Neurosci. 19, 1193–1205. doi: 10.1162/jocn.2007.19.7.1193

PubMed Abstract | CrossRef Full Text | Google Scholar

Squires, N. K., Squires, K. C., and Hillyard, S. A. (1975). Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. Electroenceph. Clin. Neurophysiol. 38, 387–401.

PubMed Abstract | Google Scholar

Stefanics, G., and Czigler, I. (2012). Automatic prediction error responses to hands with unexpected laterality: an electrophysiological study. Neuroimage 63, 253–261. doi: 10.1016/j.neuroimage.2012.06.068

PubMed Abstract | CrossRef Full Text | Google Scholar

Sutton, S., Braren, M., Zubin, J., and John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty. Science 150, 1187–1188. doi: 10.1126/science.150.3700.1187

PubMed Abstract | CrossRef Full Text | Google Scholar

Thompson, B., Perlman, M., Lupyan, G., Sehyr, Z. S., and Emmorey, K. (2020). A data-driven approach to the semantics of iconicity in American Sign Language and English. Lang. Cogn. 12, 182–202. doi: 10.1017/langcog.2019.52

CrossRef Full Text | Google Scholar

Travers, E., Friedemann, M., and Haggard, P. (2021). The Readiness Potential reflects planning-based expectation, not uncertainty, in the timing of action. Cogn. Neurosci. 12, 14–27. doi: 10.1080/17588928.2020.1824176

PubMed Abstract | CrossRef Full Text | Google Scholar

Tse, C. Y., Lee, C. L., Sullivan, J., Garnsey, S. M., Dell, G. S., Fabiani, M., et al. (2007). Imaging cortical dynamics of language processing with the event-related optical signal. PNAS 104, 17157–17162. doi: 10.1073/pnas.0707901104

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Petten, C., and Luka, B. J. (2006). Neural localization of semantic context effects in electromagnetic and hemodynamic studies. Brain Lang. 97, 279–293. doi: 10.1016/j.bandl.2005.11.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Walter, W. G., Cooper, R., Aldridge, V. J., McCallum, W. C., and Winter, A. L. (1964). Contingent Negative Variation: an electric sign of sensorimotor association and expectancy in the human brain. Nature 203, 380–384. doi: 10.1038/203380a0

PubMed Abstract | CrossRef Full Text | Google Scholar

Wei, J. H., Chan, T. C., and Luo, Y. J. (2002). A modified oddball paradigm “cross-modal delayed response” and the research on mismatch negativity. Brain Res. Bull. 57, 221–230. doi: 10.1016/S0361-9230(01)00742-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Wesselmeier, H., and Müller, H. M. (2015). Turn-taking: From perception to speech preparation. Neurosci. Lett. 609, 147–151. doi: 10.1016/j.neulet.2015.10.033

PubMed Abstract | CrossRef Full Text | Google Scholar

Woll, B. (2010). “How the brain processes languages in different modalities,” in Les lengües de signes com a lengües minoritàries: perspectives lingüístiques, socials i polítiques, eds M. J. Castell and M. J. M. Serra (Barcelona: Institut d'Estudis Catalans), 143–172.

Google Scholar

Woods, D. L., Alho, K., and Algazi, A. (1992). Intermodal selective attention. I. Effects on event-related potentials to lateralized auditory and visual stimuli. Electroencephalogr. Clin. Neurophysiol. 82, 341–355. doi: 10.1016/0013-4694(92)90004-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Xia, T., Xu, G., and Mo, L. (2019). Bi-lateralized Whorfian effect in color perception: evidence from Chinese Sign Language. J. Neurolinguistics 49, 189–201. doi: 10.1016/j.jneuroling.2018.07.004

CrossRef Full Text | Google Scholar