Julie A. Brefczynski-Lewis1,2,3 Michael E. Berrebi1 Marie E. McNeely1 Amy L. Prostko1,2 Aina Puce1,2,3*- 1 Center for Advanced Imaging, West Virginia University, Morgantown, WV, USA
- 2 Department of Radiology, West Virginia University, Morgantown, WV, USA
- 3 Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA
Facial movements have the potential to be powerful social signals. Previous studies have shown that eye gaze changes and simple mouth movements can elicit robust neural responses, which can be altered as a function of potential social significance. Eye blinks are frequent events and are usually not deliberately communicative, yet blink rate is known to influence social perception. Here, we studied event-related potentials (ERPs) elicited to observing non-task relevant blinks, eye closure, and eye gaze changes in a centrally presented natural face stimulus. Our first hypothesis (H1) that blinks would produce robust ERPs (N170 and later ERP components) was validated, suggesting that the brain may register and process all types of eye movement for potential social relevance. We also predicted an amplitude gradient for ERPs as a function of gaze change, relative to eye closure and then blinks (H2). H2 was only partly validated: large temporo-occipital N170s to all eye change conditions were observed and did not significantly differ between blinks and other conditions. However, blinks elicited late ERPs that, although robust, were significantly smaller relative to gaze conditions. Our data indicate that small and task-irrelevant facial movements such as blinks are measurably registered by the observer’s brain. This finding is suggestive of the potential social significance of blinks which, in turn, has implications for the study of social cognition and use of real-life social scenarios.
Introduction
In this digital age, the addition of blinks to the onscreen behavior of an avatar or cartoon character can make it appear “alive” and therefore is an important component of the animation process (Thomas and Johnston, 1995; Heudin, 2004). In real-life, spontaneous blinks in humans typically occur on average 20–50 times a minute (Bentivoglio et al., 1997). The considerable variation in spontaneous blinking occurs because blinking serves a number of very important non-social functions. Blinking protects the corneal surface by allowing it to be remoistened and cleaned, and will vary with individual characteristics of the tear-film or whether an individual wears contact lenses or not (Ousler et al., 2008). Blinking can also function as a defensive reflex action that not only protects the corneal surface from incoming particles and potential physical injury, but is also elicited by startle (Pearce, 2008). Aside from these protective mechanisms for the cornea, changes in blinking behavior have been shown to occur as a function of non-ocular factors. For example, decreases in blink rates have been associated with memory operations, thinking, and attentive behaviors (Holland and Tarlow, 1975; Hirokawa, et al., 2004). It is well-known that in a number of neurological disorders, such as schizophrenia, Tourette’s syndrome, Parkinson’s disease, and progressive supranuclear palsy, patients have been reported to show increased blink rates that are thought to be associated with aberrant changes to dopaminergic systems in the brain (Chan and Chen, 2004; Karson, et al., 1984; Swarztrauber and Fujikawa, 1998; Tulen et al., 1999). Hence, from a social cognitive point of view, blink rates have the potential to give clues about someone else’s inner mental life, or indeed their psychiatric well-being. Therefore, being aware of another’s blinks as well as other non-verbal behaviors might provide information that might be useful during a social interaction.
Thus far, virtually no attention has been paid to the neural consequences of observed blinks. Intuitively, it would seem that we do not notice every time someone blinks, yet we do notice when someone does not blink, or blinks very frequently. This suggests that at some level the brain does monitor the blinks of others (Omori et al., 1997). Results from behavioral studies support the idea that observed blinks are not ignored. Very frequent blinkers are more likely to be rated as nervous or careless, whereas both very infrequent and frequent blinkers are rated as “unfriendly” (Omori and Miyata, 2001). In addition, political candidates with low blink rates are rated less favorably than those with normal blink frequencies (Patterson et al., 1992). Reduced blink rate has also been linked with deception, such that criminals who lied under interrogation had lower blink rates than those who told the truth, as verified by forensic and witness evidence (Mann et al., 2002). Another study found that blink rate decreased during lying but increased considerably immediately following the lie (Leal and Vrij, 2008). The relationship of blink rate and lying may be linked to cognitive demand as it has been shown that higher demand results in less frequent blinking (Drew, 1951), followed by an increase (or flutter) when the demand is lifted (Holland and Tarlow, 1972). Overall, these studies suggest that observed blinks may have social importance and can influence higher-order processes such as making judgments about mental states or forming impressions about others.
Changes in gaze are known to elicit reliable neural responses, even when the stimulus has no overt social significance (Puce et al., 2000). The elicited event-related potentials (ERPs) mainly consist of an N170 component that can vary its amplitude as a function of whether gaze is direct or averted (Puce et al., 2000, 2003; Conty et al., 2007), followed by later ERP components that have been shown to be modulated by social attention or social significance (Sabbagh et al., 2004; Carrick et al., 2007; Conty et al., 2007). Previous studies using static faces have shown significantly greater N170 and later ERP amplitudes for stimuli with emotional or social meaning including depictions of intense emotion and personally familiar faces compared with neutral expressions or unfamiliar persons (Caharel et al., 2002; Pizzagalli et al., 2002; Eger et al., 2003).
Here we recorded neural responses elicited to viewing blinks and eye movements of other individuals. Specifically, we studied viewing eye blinks, eye closure, as well as upward and leftward gaze shifts [in previous studies no differences to viewing right and leftward gaze had been observed (Puce et al., 2000)]. As in a naturalistic situation, the blink had an extremely short duration, whereas all other eye movement conditions including eye closure were longer and matched in duration. We formulated two hypotheses relating to the ERP results of the study:
H1. That N170 and later ERP components elicited to blinks would be robust and reliable. If this prediction were to be true, it would suggest that the brain can register and process all types of eye changes for potential social relevance.
H2. That there would be a graded change in N170 and later ERP amplitude as a function of condition in the following order: averted eyes (leftward or upward), eye closure, blinks. Larger and earlier N170s and subsequent ERPs would be seen in the former conditions as they are more likely to signal social or attentional significance than the latter.
We did not expect to observe any differences in P100 amplitudes (or latency) across conditions, based on literature indicating that this ERP component is modulated by stimulus attributes such as contrast, luminance, and spatial frequency, and does not systematically vary as a function of facial attributes or movement (in particular, see a discussion of this issue in Rossion and Caharel, 2011).
Materials and Methods
Participants
Data from 14 volunteer participants between 18 and 29 years of age (mean age 21.6 years; six females; all right-handed) were included in this study. Handedness was confirmed using the Edinburgh Handedness Inventory (mean handedness quotient of 88R). All participants had normal or corrected-to-normal hearing and vision and reported no previous history of neuropsychiatric disorders. ERP data from an additional four participants were collected, but were excluded due to excessive and persistent movement or blink artifacts. Informed consent was acquired for the experiment, as approved by the Institutional Review Board of West Virginia University.
Stimuli
Stimuli consisted of gray scale video clips of a full-on face with a neutral expression and eyes looking directly at the observer. The eyes would randomly change to blink, to gaze upward and back, to gaze leftward and back, or make a duration-matched (with gaze conditions) eye closure (Figure 1). The direct gaze face was present for the entire experimental run (see also Puce et al., 2000), such that it appeared the face was looking at the observer and periodically blinking, closing the eyes, or looking up or away, always returning to the baseline direct gaze state. These apparent movements made up the four eye conditions: Blink, Eye Closure, Leftward Gaze, and Upward Gaze. All gaze aversion conditions (Eye Closure, Leftward and Upward gaze) had equal durations (100 ms) and were designed to look natural, whereas the eye blink was very brief (33 ms). The direct gaze face (no blank screen, or scrambled stimulus) remained on the screen between trials, with an interstimulus interval of 2 s between each eye change. There were two experimental runs of 4 min each, one with a male face and the other with a female face, and presentation order was counterbalanced across participants. Both faces were unfamiliar to all participants. To encourage participants to maintain their attention on the stimulus display, an additional target detection condition was included. Participants were required to detect a transparent checkerboard pattern briefly superimposed on the direct gaze face (target stimulus) and make a button press response each time it appeared. There were a total of 40 trials for each eye movement condition, as well as 40 target trials (half of these occurred with the female face, the other half with the male face).
Figure 1. Examples of apparent motion face stimulus conditions. Display changes were embedded within a continuously present Baseline condition, and these task irrelevant gaze changes were Blink, Eye Closure, Upward, Leftward. A checkerboard pattern superimposed upon the baseline condition face was presented infrequently and served as the Target stimulus. The resulting display produced an apparent motion effect where the eyes on the face were altered across the various conditions which each had a 100 ms duration, with the exception of the Blink whose duration was 33 ms. The example depicts the female face. A male face was also used in the experiment.
All video stimuli were presented using Presentation software (version 11; Neurobehavioral Systems, CA, USA). Video clips of the faces were dynamic and featured apparent motion of the eyes, which maintained their end positions for 100 ms (Leftward gaze, Upward gaze, and Eye Closure) or 33 ms (Blink) before returning to the baseline direct gaze condition. To maintain consistent stimulus timing, target duration (superimposed checkerboard pattern) was also equal to the duration of the presented eye change stimulus. Although there were only three video frame changes from direct gaze to eye change back to direct gaze in this apparent motion stimulus, as in a previous study (Puce et al., 2000), all participants described the eye movements as realistic. The stimulus faces were projected on a white screen placed directly in front of the participant at a distance of 195 cm, and subtended a visual angle of 14.3° × 16.7° (horizontal × vertical). The two eyes in each face subtended a visual angle of 3.4° × 0.73°. Stimulus condition order was randomized across trials.
Data Collection
Each participant was tested individually in a quiet, dimly lit room, while seated in an armchair. Visual stimuli were projected onto a white screen in front of the participant rather than viewed from a computer monitor to improve visibility of the facial movements and make the face seem more realistic (projected dimensions were approximately life size). Each participant completed two experiments in one ERP session in randomized order. Participants were given a short rest period (several minutes) between experiments. The second experiment had an unrelated purpose and will not be reported here. Before the experiment, participants were instructed to minimize their own eye and face movements, to keep their gaze centered on the bridge of the nose of the stimulus face, and to press the response button to targets as quickly and accurately as possible.
ERP Recording
The electroencephalogram (EEG) was recorded from participants using a 128 channel Neuroscan Electrocap with sintered Ag/AgCl electrodes, with a midline frontal ground and dual reference electrodes placed on either side of the nose. Four electro-oculographic (EOG) electrodes captured horizontal and vertical EOG components (Figure A1 in Appendix). EEG and EOG signals were recorded with a gain of 5000, sampled at 250 Hz/channel, and band pass filtered at 0.1–100 Hz. A continuous EEG recording was made during each experimental run and included a digital time stamp to indicate the onset of each stimulus condition.
At the completion of the recording session, electrode locations were digitized with a Polhemus 3 Space Fastrak digitizer (see Appendix).
ERP Analysis
Electroencephalogram data were segmented offline into epochs of 1100 ms, which included a 100 ms pre-stimulus baseline. A semi-automated artifact detection routine identified participant eye blinks in segmented EEG epochs, and data were also visually inspected for additional electromyographic artifacts. Artifactual EEG epochs were excluded from subsequent analyses. An average of 10.6 out of 40 trials were excluded per participant per condition. There were no significant differences in artifact removal between conditions (average for viewing Blink 10.6, Eye Closure 10.7, Upward gaze 9.6, Leftward gaze 11.2 trials). Segmented EEG epochs were low pass filtered using a filter cut-off of 40 Hz, then baseline corrected with respect to the pre-stimulus baseline, and averaged as a function of each stimulus condition for each participant. ERPs elicited in response to target stimuli were excluded from further analyses.
The averaged ERPs for each participant were used to create grand average ERPs for each stimulus type across the participant group. Potential latency ranges for a subsequent semi-automated peak identification analysis were defined for each ERP component from the grand averaged ERPs waveforms. Peak latencies and amplitudes for all stimulus types in individual participants and grand averaged ERPs were identified within those pre-selected time intervals. From the average ERPs of each individual participant, ERP component peak latency, and amplitude measures were calculated for each condition.
Topographical voltage maps and Laplacians (second spatial derivative of the voltage map; Puce et al., 2007) of the grand average ERPs at ERP component peaks were created using EMSE software (Source Signal Systems, CA, USA).
Based on inspection of topographic voltage maps, clusters of electrodes were selected whose mean ERP amplitude and latency values were calculated for P100, N170, and amplitudes for later ERP components. Data for P100 and N170 components were entered into two-way repeated measures analysis of variance (ANOVA) using SPSS V15 to test for Condition and Hemisphere. Amplitude data for later ERP components were analyzed using a repeated measures ANOVA for Condition.
Results
Behavioral Data
Participants indicated by keyboard press when they detected a transparent black and white checkerboard overlaid on the direct gaze face. The mean reaction time for target detection was 501.0 ms (SD = 138.6 ms), with 95.1% accuracy.
ERP Data
All eye change conditions, including the viewed eye blink, produced a robust ERP complex consisting of a P100 and N170 seen mainly at temporo-occipital electrodes (e.g., centered on equivalent 10–10 system sites P07 and P08). Additionally, large later ERP components, were also observed to all stimulus conditions, and these later ERPs tended to be more extensively distributed across the scalp (see white markers on topographic maps in Figure 3). We discuss each ERP component below.
Early ERP Components at Temporo-Occipital Electrodes
N170
As we predicted (H1), the Blink condition produced a large and reliable N170 (see Figure 2A); however, all eye conditions produced early components of seemingly equal amplitude (Figure 2A, right temporo-occipital scalp waveforms for Blink and Leftward gaze are shown as examples) over the temporo-occipital scalp, contrary to our hypothesis (H2). The N170 was greater in spatial extent for the right hemisphere. Our second prediction (H2), that we would observe a gradient in amplitude as function of stimulus condition (Leftward and Upward, Eye Closure, Blink) was not sustained for the N170 (see Figures 2B,C histograms).
Figure 2. Group data: P100 and N170. (A) P100 and N170 components for the Blink and Leftward gaze conditions are seen in group averaged ERP waveforms generated from a compound average at six electrodes each at the left and right occipito-temporal scalp [see white dots in (B,C)]. Red line indicates the onset of an eye change condition. (B) P100 and (C) N170 topographic voltage maps and amplitude histograms for Blink and Leftward gaze across all subjects. Left panels: voltage topographic maps at the peak activity for P100 and N170 for the Blink and Leftward gaze conditions (amplitude color scale is located at the left of the maps). White dots depict electrodes included in the averaged waveforms displayed in (A) and used in statistical analyses, whereas black dots denote the positions of the other recording electrodes. Right panel: P100 and N170 amplitude histograms for all four eye movement conditions at each cluster of electrodes over the posterior temporal scalp.
Statistical testing of N170 amplitude differences between conditions via a two-way repeated measures ANOVA for Condition and Hemisphere, was performed on the N170 peak amplitude from an averaged ERP from a cluster of six temporo-occipital electrodes in each hemisphere (see Figures 2B,C white dots; and see Puce et al., 2003; Carrick et al., 2007). There was no main effect of stimulus Condition (Table 1), however, there was a significant main effect of Hemisphere. Comparisons indicated that the right hemisphere generated larger and slower N170s in all conditions (Figure 2C; Table 1). Similar to N170 amplitudes, a statistical analysis of N170 latency revealed a main effect of Hemisphere, but no main effect of Condition, and no significant interaction effects (data not shown).
Table 1. Summary of repeated measures ANOVA for ERP component amplitudes as a function of Condition and Hemisphere.
P100
A two-way ANOVA revealed no main effect or interaction effect of Condition or Hemisphere for P100 amplitude at temporo-occipital electrodes (Figure 2B). There was no significant main effect of latency, nor were there any significant interaction effects.
Late ERP Components
To test the hypothesis that late ERP components would show an amplitude gradient based on putative social significance, we examined later ERPs at clusters of electrodes showing the greatest amplitudes. A series of positive late ERPs were observed, occurring at around 250, 450, and 600 ms post-stimulus, i.e., P250, P450, P600. To facilitate distinction of spatial distributions of the later ERPs, Figure 3 displays Upward and Leftward gaze topographic voltage maps with higher voltage thresholds than those for Eye Closure and Blink (Figures 3A–C; for Leftward gaze at same threshold see Figure A2 in Appendix). In terms of spatial distribution and extent, the scalp distributions of the later ERP components showed some differences between the Blink and the other conditions (e.g., Figure 3). We discuss each of these ERP components separately.
Figure 3. Group data: late ERPs. Topographic voltage maps for the Blink, Eye Closure (Closed), Upward and Leftward gaze conditions displayed, with Upward and Leftward gaze (right middle) displayed using a larger voltage scale. White dots on voltage topographic maps indicate electrodes used in ANOVAs and in charts (at right). Gray stars in (B) indicate the peak electrodes (anterior temporal) for the Blink condition P450. Amplitude charts show all four conditions (A) P250, (B) P450, and (C) P600. Significant differences on the charts are not indicated due to complexity, but are listed in the text.
P250
A clear P250 (mean latency 263 ms), which generally peaked over the vertex (Figure 3A, e.g., Leftward gaze condition), was elicited by all four eye movement conditions, but was more posterior (centered closer to Pz) for the Blink condition relative to the other conditions (Figure 3A, compare to Leftward gaze condition). A one-way ANOVA (Condition) of peak amplitudes from a five electrode cluster centered on Cz showed a main effect of Condition. This effect was driven by the Blink and Eye Closure conditions which both showed significantly smaller amplitudes than either the Upward or Leftward gaze conditions (Table 1). Note, however, that the Blink and Eye Closure conditions did produce significant peaks at this latency (Figure 3, charts on right). An ANOVA for P250 latency indicated that there were no significant effects across condition.
P450
All four eye movement conditions produced a significant P450 (mean latency 459 ms). P450 was seen over the anterior temporal scalp and tended to be right lateralized for most conditions, including the Blink. The Leftward gaze aversion showed a more posterior temporal distribution than the Blink. We investigated this distributional difference further by plotting topographic Laplacian maps at the peak of P450 activity. For all conditions, there appeared to be a prominent sink and source of activity on the midline and the right posterior temporal scalp, respectively (Figure 4, bottom row). The Laplacian map for the Blink condition showed these same loci and an additional sink over the occipital scalp. This may explain the apparent hemispheric asymmetry in the topographic voltage maps for this condition. In the left hemisphere a sink of activity was also observed in the temporal scalp with the Eye Closure, Upward, and Left conditions, and in the fronto-temporal scalp for the Blink condition.
Figure 4. Comparison of Voltage vs. Laplacian Topographic maps at the peak latency of P450. Voltage maps (top row) and Laplacian maps (bottom row) are shown for all four eye movement conditions, with threshold matched across conditions. Positive amplitudes/sources are warm colors; Negative amplitudes/sinks are cool colors as shown in the colored scale bar.
Given the observed distribution differences in P450 across conditions, we performed statistical analyses for two electrode clusters centered on both these amplitude maxima (anterior temporal and posterior temporal). A two-way ANOVA for Condition and Hemisphere of the posterior temporal electrode amplitudes (white circles in Figure 3B), revealed a significant effect of Condition with Leftward gaze having a significantly greater amplitude than the three other conditions (Table 1). There was no main effect of Hemisphere and no interaction effect. On the other hand, the anterior temporal electrodes (gray stars in Figure 3B), showed main effects of Condition and Hemisphere (Table 1). The Condition effect was driven by larger amplitudes for the gaze conditions. The Hemisphere effect was driven by significantly greater amplitudes in the right vs. left hemisphere for Blink, Eye Closure, and Upward gaze conditions (all comparisons included in Table 1). For P450 latency, no significant differences in the two-way ANOVA for Condition or Hemisphere for either the anterior temporal or posterior temporal electrode cluster.
P600
On average, all of the eye movement conditions also produced a broad positivity at around 600 ms (P600, mean latency 623 ms). Again there was a difference in the location of the maximal amplitude for this component, with Leftward gaze showing a more anterior focus compared to Blink (see Figure 3C; no difference in Condition effect for Pz vs. Cz electrodes, data not shown). At a nine electrode cluster centered on parietal midline electrode Pz, there was a main effect of Condition with P600s in response to Leftward gaze having significantly greater amplitudes than the three other conditions (Table 1).
Duration of Significantly Different Neural Activity as a Function of Time
Finally, we examined the data of individual participants for potential differences in the duration of significant activation across the Blink and Leftward gaze conditions. We did this over multiple electrodes which included locations over the right occipito-temporal, midline parietal, central, frontal, and right anterior frontal scalp (Figure 5). To do this, we measured the time elapsed after which no significant peaks occurred; an effective return to baseline (RTB). To determine the RTB we first calculated the t-value vs. zero for each point on the waveform (using Neuroscan Scan 4.3 software). The RTB was defined as the end of the last significant peak of sustained (20 ms, 5 time points) significance (t > 1.67 for n = 22–46 epochs, see Appendix for details). The plot in Figure 5 displays time interval post-stimulus on the y-axis and individual participant data on the x-axis. A group mean is also depicted. A right temporo-occipital electrode, analogous to PO8 in the 10–10 system, which typically exhibits a maximum for N170 amplitude, showed that for both Blink and Leftward gaze, every participant exhibited a significant N170 peak (Figure 5, see gray overlay). Interestingly, not all individuals showed later ERP activity. For other midline electrodes (Cz, Pz, Fpz), more participants showed late peaks for the Leftward gaze condition compared to the Blink condition (e.g., for P250 at Cz: 14 for Leftward, 10 for Blink; for P600 at Pz: 13 for Leftward, 9 for Blink; for late frontal at Fpz: 11 for Leftward, 8 for Blink).
Figure 5. Return to Baseline (RTB) times in individual participants. Blink (top row) and Leftward gaze (bottom row) condition data are shown for each participant as bars 1–14. The RTB times indicate a permanent RTB (time in ms). The gray bar for each participant extends for the duration of time in which significant peaks (vs. baseline) are observed (timeline on y-axis). The criterion for a waveform to be significantly different from zero was five consecutive time points (20 ms) with a t-value greater than 1.68 (based on a df = 40, p < 0.05). All participants had significant early peaks at temporo-occipital electrodes (up to ∼200 ms, see gray shaded region for P08), but not all participants had later peaks, or peaks at more anterior electrodes, especially for the Blink condition. The final bar indicates a group average RTB with SE. RTB values are shown for electrodes analogous to 10–10 system sites PO8, Cz, Pz, and Fpz.
In sum, for late ERPs, contrasts revealed a general trend in progressively decreasing ERP amplitudes, in that Leftward ≥ Upward > Closure = Blink. There were also laterality differences and scalp distribution differences across conditions. Although fewer individual participants showed significant late potentials for the Blink condition, on average, there were significant potentials, even at 600 ms. The P450 at the posterior temporal electrodes and the P600 at the midline parietal scalp for Leftward gaze, the condition with the most potential social and attentional meaning, had significantly greater amplitudes than all the other conditions.
Discussion
N170 is Comparable for All Tested Eye Movement Conditions
Our second hypothesis that the Blink condition would elicit reduced N170 amplitudes, compared to the other conditions, was not sustained; robust N170s were elicited by all conditions including Blink. Similarly, no significant differences were seen in N170 latency as a function of Condition. We had previously described significant differences in N170 amplitudes and latencies as a function of eye gaze direction – with larger N170 observed to gaze aversions relative to when eyes look directly at the observer (Puce et al., 2000, 2003). Type of gaze aversion (left, right, upward) does not appear to influence N170 amplitude, irrespective of whether an explicit social context or social judgment is required or not. As expected our tested gaze aversions (Leftward, Upward) also showed no significant differences in N170 amplitude or latency, however, in this study we used transient aversions of around 100 ms in an attempt to move toward more ecologically valid stimulus displays. Here we also tested Eye Closure, which might be regarded as a form of eye aversion, using a similar stimulus duration of 100 ms, in addition to a Blink – a quicker type of eye closure stimulus. In these latter two stimulus conditions, N170 amplitudes were as large as for the gaze aversions, indicating that the brain registers and monitors all types of eye changes. The N170 has been previously suggested to potentially be sensitive to changes in the social attention of another individual (Itier and Batty, 2009), but the results in this study would suggest that N170 might more likely reflect a detector of change in the eyes, or perhaps changes in the face in general (see also Puce et al., 2000).
There was an effect of Hemisphere for the N170, with the right hemisphere showing significantly greater amplitude than the left hemisphere in all four conditions. In our own previous studies, N170s elicited to viewing lateral gaze shifts to both the left and right were of equal magnitude, with a tendency for larger N170s to be seen in the right hemisphere regardless of the gaze shift direction (Puce et al., 2000). The right hemisphere preference for face related stimuli, be they the onset of static faces or dynamic faces in ongoing facial displays, has been observed in studies by other laboratories (Rossion et al., 1999; Itier and Batty, 2009). The existing literature, which is devoted largely to examining N170s elicited by static face onset, shows mixed findings with respect to facial changes related to affect or social significance. For example, some studies showed larger N170 amplitudes to static fearful vs. neutral face onset (Pizzagalli et al., 2002; Eger et al., 2003), while others showed no differences (Eimer and Holmes, 2002).
We also examined the behavior of the P100, an ERP component which is thought to reflect low-level stimulus attributes in faces as well as other stimulus categories (see Rossion and Caharel, 2011). We did not observe any differences in P100 attributes as a function of stimulus Condition. Our data indicate that differentiation of neural activity as a function of taking the eyes away from another only occurs at latencies beyond 250 ms (see below). It is our belief that these earlier ERPs signal the change in the facial display (P100, N170), and potentially socially salient stimuli relative to the observer (N170). This latter signal may encourage further attention to a stimulus for potential social or attentional relevance.
Late ERP Signals
Average amplitudes of late ERP components were greater for Leftward and Upward gaze aversions compared to Blinks and Eye Closure, although clear and discernable late peaks were observed even in these latter two conditions. This was true for P250, P450, and P600, supporting our hypothesis that late potential amplitudes may reflect attentional and/or social significance, as seen in a previous study, where social context was explicitly manipulated (Carrick et al., 2007). Traditionally, later ERPs have been associated with higher-order cognitive and affective processes (Sutton et al., 1967; Kutas et al., 1977; Kutas and Hillyard, 1980; Hillyard and Kutas, 1983; Rosler et al., 1986), and more recently have been described in social cognition and theory of mind tasks (Liu et al., 2004; Sabbagh et al., 2004; Puce et al., 2007). Therefore the later responses observed in this study may indicate that simple eye movements, including blinks, have some social importance to us.
Event-related potential studies using static face onsets suggest that P250 may be related to consolidating configural information of faces (Halit et al., 2000; Milivojevic et al., 2003), while ERPs beyond 350 ms may be related to assessment of stimuli, social cognition, memory, and attentional processes (Cacioppo et al., 1993; Sabbagh et al., 2004). In addition, later ERP amplitudes are known to increase with attentional or emotional salience (Talsma and Woldorff, 2005; Schupp et al., 2007). A recent study from our lab using dynamic facial displays showed differences in later ERPs to averted gaze when embedded in multiface displays where an explicit judgment of social significance is required (Carrick et al., 2007). Even in the present study in which participants did not make explicit social judgments, these changes in later ERPs were robust and reliable. Perhaps this is because, in everyday social interactions, seeing an upward or lateral gaze shift is usually a strong indicator that the attention of the observed person is being directed away from the observer, conceivably toward important external people or stimuli, or as a result of social discomfort (Kleinke, 1986).
It is possible that as we observe the eye movements of another, including blinks, we may activate an embodied response within the oculomotor system. Oscillatory changes for self-blinks in occipital regions at 200 ms and in orbital regions at 400 ms have been documented with MEG (Bardouille et al., 2006). Furthermore, fMRI studies of voluntary self-blinks show activation in visual and orbital frontal cortices (Tsubota et al., 1999; Yoon et al., 2005). For self-blinks, however, these changes might also reflect a transient interruption of input to visual cortex. Activation in different cortical structures may help explain the distributional differences observed in later ERP components across conditions. Examples are the more posteriorly distributed P250 and P600, and more anterior and right lateralized P450 for the Eye Closure vs. Leftward and Upward gaze change conditions. Future studies using neural source modeling might be able to shed some light on these differences.
Future Directions
Given that we appear to be aware of the blink rates of others (Omori and Miyata, 2001; Mann et al., 2002) and that social perception is known to be influenced by blink rates of the observed person, a study in which another’s blinking behavior was explicitly associated with particular social attributes might be a way to systematically investigate how late ERPs are modulated by these social attributes. Correlating the degree of higher-order blink processing with blink rate (normal, fast, or slow) and personality assessments would be useful in understanding how non-verbal cues can influence our judgments. A tantalizing possibility for why blinks result in higher-order cognitive processing is that we have learned associations between blink rate and level of interest, arousal level, stress and/or other factors integral to social interactions. These cognitive and affective states can in turn affect dopaminergic systems which have a direct relationship with blink rate (Brozoski et al., 1979; Karson, 1983; Coull et al., 1995; Coull, 1998; Hirokawa et al., 2000; Dreisbach et al., 2005).
Reflexive attentional shifts that result from observed non-predictive (and sometimes anti-predictive) eye gaze changes (Friesen et al., 2004; Schuller and Rossion, 2005) might potentially drive the observed differences in later ERPs in our study. For example, in the current study, if objects or another face or person were located above or to the left of the face, attention may have shifted to these objects/people causing differences in attentional focus, ERPs, and distraction level (Fichtenholtz et al., 2007). Adding affective expressions to the gaze changes would likely create even more complex attentional effects, which might be influenced by such factors if the depicted emotion encourages approach or withdrawal on the part of the observer (Fichtenholtz et al., 2007, 2009).
Caveats
Our task used an apparent motion task with two-dimensional real human faces and near-realistic timing. However, there can be differences in neural responses to viewing our stimuli vs. observing live interactions (Ponkanen et al., 2008). The higher-order response to blinks observed in our study could conceivably occur as a result of the artificial and isolated nature of our computer-presented stimuli. It is possible that in this relatively impoverished laboratory environment, less important stimuli may be processed to a greater extent than they would normally be in a natural environment. In addition, multiple repetitions of the limited stimulus set may have led to habituation, which could have in turn have led to reduced higher-order processing of all eye movements. However, we believe this to be unlikely, as significant differences between conditions were seen in later ERPs and not the early components.
Our sample size (14 subjects) may have resulted in the absence of significant differences in the N170 amplitudes across conditions, although in our previous studies using this sample size, significant differences in N170 amplitudes were observed when social attention was manipulated.
Lastly, there was a stimulus duration difference between the Blink condition and the other conditions. However, the Eye Closure condition was matched in duration with the Leftward and Upward gaze conditions. Eye Closure did not show significant differences from Blink, and, like the Blink condition, was often significantly smaller than the Leftward gaze condition. However, a shorter gaze change condition may be necessary to fully address this duration difference.
Conclusion
Behavioral studies indicate that we form social perceptions about others from observing their behaviors, which would include monitoring blinking behavior. Our ERP data show that blinks are registered similarly to gaze aversions, the latter of which are known to be important for gauging the social attention of another. Importantly, later ERP components were sensitive to the potential social/attentional significance of the stimuli and displayed a social relevance related response gradient. Our work demonstrates that blinks in and of themselves may prove to be interesting and important cognitive events.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
Julie Brefczynski-Lewis and Aina Puce are supported by R01 NS049436 grant awarded by the NINDS. Michael Berrebi and Marie McNeely were supported by a Summer Undergraduate Research Internship (SURI) awarded by the Center for Neuroscience at West Virginia University. We thank Ms. Mary Pettit for editorial assistance. Ms. Pettit is supported by the Radiology Department at West Virginia University.
References
Bardouille, T., Picton, T. W., and Ross, B. (2006). Correlates of eye blinking as determined by synthetic aperture magnetometry. Clin. Neurophysiol. 117, 952–958.
Bentivoglio, A. R., Bressman, S. B., Cassetta, E., Carretta, D., Tonali, P., and Albanese, A. (1997). Analysis of blink rate patterns in normal subjects. Mov. Disord. 12, 1028–1034.
Brozoski, T. J., Brown, R. M., Rosvold, H. E., and Goldman, P. S. (1979). Cognitive deficit caused by regional depletion of dopamine in prefrontal cortex of rhesus monkey. Science 205, 929–932.
Cacioppo, J. T., Crites, S. L. J., Berntson, G. G., and Coles, M. G. H. (1993). If attitudes affect how stimuli are processed, should they not affect the event-related brain potential? Psychol. Sci. 4, 108–112.
Caharel, S., Poiroux, S., Bernard, C., Thibaut, F., Lalonde, R., and Rebai, M. (2002). ERPs associated with familiarity and degree of familiarity during face recognition. Int. J. Neurosci. 112, 1499–1512.
Carrick, O. K., Thompson, J. C., Epling, J. A., and Puce, A. (2007). It’s all in the eyes: neural responses to socially significant gaze shifts. Neuroreport 18, 763–766.
Chan, R. C., and Chen, E. Y. (2004). Blink rate does matter: a study of blink rate, sustained attention, and neurological signs in schizophrenia. J. Nerv. Ment. Dis. 192, 781–783.
Conty, L., N’Diaye, K., Tijus, C., and George, N. (2007). When eye creates the contact! ERP evidence for early dissociation between direct and averted gaze motion processing. Neuropsychologia 45, 3024–3037.
Coull, J. T. (1998). Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology. Prog. Neurobiol. 55, 343–361.
Coull, J. T., Sahakian, B. J., Middleton, H. C., Young, A. H., Park, S. B., McShane, R. H., Cowen, P. J., and Robbins, T. W. (1995). Differential effects of clonidine, haloperidol, diazepam and tryptophan depletion on focused attention and attentional search. Psychopharmacology (Berl.) 121, 222–230.
Dreisbach, G., Muller, J., Goschke, T., Strobel, A., Schulze, K., Lesch, K. P., and Brocke, B. (2005). Dopamine and cognitive control: the influence of spontaneous eyeblink rate and dopamine gene polymorphisms on perseveration and distractibility. Behav. Neurosci. 119, 483–490.
Drew, G. C. (1951). Variations in reflex blink-rate during visual motor tasks. Q. J. Exp. Psychol. 3, 73–88.
Eger, E., Jedynak, A., Iwaki, T., and Skrandies, W. (2003). Rapid extraction of emotional expression: evidence from evoked potential fields during brief presentation of face stimuli. Neuropsychologia 41, 808–817.
Eimer, M., and Holmes, A. (2002). An ERP study on the time course of emotional face processing. Neuroreport 13, 427–431.
Fichtenholtz, H. M., Hopfinger, J. B., Graham, R., Detwiler, J. M., and Labar, K. S. (2007). Happy and fearful emotion in cues and targets modulate event-related potential indices of gaze-directed attentional orienting. Soc. Cogn. Affect. Neurosci. 2, 323–333.
Fichtenholtz, H. M., Hopfinger, J. B., Graham, R., Detwiler, J. M., and LaBar, K. S. (2009). Event-related potentials reveal temporal staging of dynamic facial expression and gaze shift effects on attentional orienting. Soc. Neurosci. 4, 317–331.
Friesen, C. K., Ristic, J., and Kingstone, A. (2004). Attentional effects of counterpredictive gaze and arrow cues. J. Exp. Psychol. Hum. Percept. Perform. 30, 319–329.
Halit, H., de Haan, M., and Johnson, M. H. (2000). Modulation of event-related potentials by prototypical and atypical faces. Neuroreport 11, 1871–1875.
Heudin, J. (2004). “Evolutionary virtual agent,” in Proceedings of the IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’04), Beijing.
Hillyard, S. A., and Kutas, M. (1983). Electrophysiology of cognitive processing. Annu. Rev. Psychol. 34, 33–61.
Hirokawa, K., Yagi, A., and Miyata, Y. (2000). An examination of effects of linguistic abilities on communication stress during a telephone situation measured by blink and heart rates. Soc. Behav. Pers. Int. J. 28, 343–354.
Hirokawa, K., Yagi, A., and Miyata, Y. (2004). Comparison of blinking behavior during listening to and speaking in Japanese and English. Percept. Mot. Skills 98, 463–472.
Itier, R. J., and Batty, M. (2009). Neural bases of eye and gaze processing: the core of social cognition. Neurosci. Biobehav. Rev. 33, 843–863.
Karson, C. N. (1983). Spontaneous eye-blink rates and dopaminergic systems. Brain 106(Pt 3), 643–653.
Karson, C. N., Burns, R. S., LeWitt, P. A., Foster, N. L., and Newman, R. P. (1984). Blink rates and disorders of movement. Neurology 34, 677–678.
Kutas, M., and Hillyard, S. A. (1980). Event-related brain potentials to semantically inappropriate and surprisingly large words. Biol. Psychol. 11, 99–116.
Kutas, M., McCarthy, G., and Donchin, E. (1977). Augmenting mental chronometry: the P300 as a measure of stimulus evaluation time. Science 197, 792–795.
Liu, D., Sabbagh, M. A., Gehring, W. J., and Wellman, H. M. (2004). Decoupling beliefs from reality in the brain: an ERP study of theory of mind. Neuroreport 15, 991–995.
Mann, S., Vrij, A., and Bull, R. (2002). Suspects, lies, and videotape: an analysis of authentic high-stake liars. Law Hum. Behav. 26, 365–376.
Milivojevic, B., Clapp, W. C., Johnson, B. W., and Corballis, M. C. (2003). Turn that frown upside down: ERP effects of thatcherization of misorientated faces. Psychophysiology 40, 967–978.
Omori, Y., and Miyata, Y. (2001). Estimates of impressions based on frequency of blinking. Soc. Behav. Pers. 29, 159–167.
Omori, Y., Yamada, F., and Miyata, Y. (1997). Influences of blinking on person perception. Jpn. J. Soc. Psychol. 83, 591–594.
Ousler, G. W., Hagberg, K. W., Schnindelar, M., Welch, D., and Abelson, M. B. (2008). The ocular protection index. Cornea 27, 509–513.
Patterson, M. L., Churchill, M. E., Burger, C. K., and Powell, J. L. (1992). Verbal and nonverbal modality effects on impressions of political candidates: analysis from the 1984 presidential debates. Commun. Monogr. 59, 231–242.
Pizzagalli, D. A., Lehmann, D., Hendrick, A. M., Regard, M., Pascual-Marqui, R. D., and Davidson, R. J. (2002). Affective judgments of faces modulate early activity (approximately 160 ms) within the fusiform gyri. Neuroimage 16, 663–677.
Ponkanen, L. M., Hietanen, J. K., Peltola, M. J., Kauppinen, P. K., Haapalainen, A., and Leppanen, J. M. (2008). Facing a real person: an event-related potential study. Neuroreport 19, 497–501.
Puce, A., Epling, J. A., Thompson, J. C., and Carrick, O. K. (2007). Neural responses elicited to face motion and vocalization pairings. Neuropsychologia 45, 93–106.
Puce, A., Smith, A., and Allison, T. (2000). ERPs evoked by viewing facial movements. Cogn. Neuropsychol. 17, 221–239.
Puce, A., Syngeniotis, A., Thompson, J. C., Abbott, D. F., Wheaton, K. J., and Castiello, U. (2003). The human temporal lobe integrates facial form and motion: evidence from fMRI and ERP studies. Neuroimage 19, 861–869.
Rosler, F., Sutton, S., Johnson, R. Jr., Mulder, G., Fabiani, M., Gorsel, E. P., and Roth, W. T. (1986). Endogenous ERP components and cognitive constructs. A review. Electroencephalogr. Clin. Neurophysiol. Suppl. 38, 51–92.
Rossion, B., and Caharel, S. (2011). ERP evidence for the speed of face categorization in the human brain: disentangling the contribution of low-level visual cues from face perception. Vision Res. 51, 1297–1311.
Rossion, B., Delvenne, J. F., Debatisse, D., Goffaux, V., Bruyer, R., Crommelinck, M., and Guerit, J. M. (1999). Spatio-temporal localization of the face inversion effect: an event-related potentials study. Biol. Psychol. 50, 173–189.
Sabbagh, M. A., Moulson, M. C., and Harkness, K. L. (2004). Neural correlates of mental state decoding in human adults: an event-related potential study. J. Cogn. Neurosci. 16, 415–426.
Schuller, A. M., and Rossion, B. (2005). Spatial attention triggered by eye gaze enhances and speeds up visual processing in upper and lower visual fields beyond early striate visual processing. Clin. Neurophysiol. 116, 2565–2576.
Schupp, H. T., Stockburger, J., Codispoti, M., Junghofer, M., Weike, A. I., and Hamm, A. O. (2007). Selective visual attention to emotion. J. Neurosci. 27, 1082–1089.
Sutton, S., Tueting, P., Zubin, J., and John, E. R. (1967). Information delivery and the sensory evoked potential. Science 155, 1436–1439.
Swarztrauber, K., and Fujikawa, D. G. (1998). An electroencephalographic study comparing maximum blink rates in schizophrenic and nonschizophrenic psychiatric patients and nonpsychiatric control subjects. Biol. Psychol. 43, 282–287.
Talsma, D., and Woldorff, M. G. (2005). Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. J. Cogn. Neurosci. 17, 1098–1114.
Thomas, F., and Johnston, O. (1995). The Illusion of Life: Disney Animation. New York: Hyperion Press.
Tsubota, K., Kwong, K. K., Lee, T. Y., Nakamura, J., and Cheng, H. M. (1999). Functional MRI of brain activation by eye blinking. Exp. Eye Res. 69, 1–7.
Tulen, J. H., Azzolini, M., de Vries, J. A., Groeneveld, W. H., Passchier, J., and van De Wetering, B. J. (1999). Quantitative study of spontaneous eye blinks and eye ticks in Gilles de la Tourette’s syndrome. J. Neurol. Neurosurg. Psychiatr. 67, 800–802.
Keywords: eye movements, faces, social cognition, eye blinks, ERPs
Citation: Brefczynski-Lewis JA, Berrebi ME, McNeely ME, Prostko AL and Puce A (2011) In the blink of an eye: neural responses elicited to viewing the eye blinks of another individual. Front. Hum. Neurosci. 5:68. doi: 10.3389/fnhum.2011.00068
Received: 27 January 2011;
Accepted: 14 July 2011;
Published online: 08 August 2011.
Edited by:
Neal J. Cohen, University of Illinois, USAReviewed by:
Jennifer D. Ryan, Rotman Research Institute, CanadaJoel Voss, University of Illinois Urbana-Champaign, USA
Copyright: © 2011 Brefczynski-Lewis, Berrebi, McNeely, Prostko and Puce. This is an open-access article subject to a non-exclusive license between the authors and Frontiers Media SA, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and other Frontiers conditions are complied with.
*Correspondence: Aina Puce, Department of Psychological and Brain Sciences, Indiana University, 1101 E 10th Street, Bloomington, IN 47405, USA. e-mail: ainapuce@indiana.edu