Working Memory Maintenance Modulates Serial Dependence Effects of Perceived Emotional Expression

Mei, Gaoxing; Chen, Shiyu; Dong, Bo

doi:10.3389/fpsyg.2019.01610

ORIGINAL RESEARCH article

Front. Psychol., 11 July 2019

Sec. Perception Science

Volume 10 - 2019 | https://doi.org/10.3389/fpsyg.2019.01610

Working Memory Maintenance Modulates Serial Dependence Effects of Perceived Emotional Expression

$\r\nGaoxing Mei*&#x;$ Gaoxing Mei^1*†

Shiyu Chen^1†

Bo Dong²

¹School of Psychology, Guizhou Normal University, Guiyang, China
²Department of Psychology, School of Education and Public Administration, Suzhou University of Science and Technology, Suzhou, China

The stability of face perception is vital in interpersonal interactions. Recent studies have revealed the mechanism of the stability in the perception of stable attributes of faces (such as facial identity) by serial dependence, a phenomenon in which perception of current stimuli is pulled toward recently viewed stimuli. However, whether serial dependence of perceived emotional expression (a changeable attribute of faces) exists remains controversial, and its exact nature has not been examined yet. To address these issues, we used the methods of constant stimuli and two-interval forced choice tasks in three psychophysical experiments. Participants compared two successive facial expressions selected from a continuum with 50 morphed faces ranging from sad to happy. Experiment 1a and 1b showed that a perceived facial expression pulled toward previously seen facial expressions (i.e., a significant serial dependence effect), independent of response instructions. Furthermore, a stronger serial dependence effect was found when the first facial expression was retained in working memory for a longer delay duration (Experiment 2), and yet a weaker serial dependence effect was observed when a longer delay between decision and response was performed (Experiment 3). These findings indicate that serial dependence facilitates the stability of facial expression perception and is modulated by working memory representations.

Introduction

Past visual experiences shape our current perception toward the visual world. On the one hand, it is well established that the visual system maintains high sensitivity to changes in the environment through visual adaptation, a phenomenon in which prolonged exposure to visual stimuli leads to negative aftereffects (for reviews, see Kohn, 2007; Webster, 2011, 2015). For example, when exposed to a sad facial expression for several seconds, observers perceived current faces as less sad (Hsu and Young, 2004; Ying and Xu, 2017; Palermo et al., 2018). On the other hand, recent studies have demonstrated that the visual system also maintains a stable and continuous perception in the noisy visual environment over time through serial dependence, a phenomenon in which the perception of current stimuli is pulled toward recently viewed stimuli (i.e., positive aftereffects) (Fischer and Whitney, 2014; Liberman et al., 2014; Kiyonaga et al., 2017). For example, perceived orientations of gratings were systematically biased toward recently viewed orientations in an orientation judgment task (Fischer and Whitney, 2014).

Serial dependence effects have frequently been reported for basic visual attributes, such as orientation (Fischer and Whitney, 2014; Liberman et al., 2016; Manassi et al., 2017), numerosity (Corbett et al., 2011; Cicchini et al., 2014; Fornaciai and Park, 2018), motion (Alais et al., 2017), size (Alexi et al., 2018), and position (Papadimitriou et al., 2015; Papadimitriou and Snyder, 2017; Manassi et al., 2018). Serial dependence has been considered a reflection of the Continuity Field: a spatiotemporal region where similar visual stimuli are integrated (Fischer and Whitney, 2014; Liberman et al., 2018).

Human faces, as naturally complex stimuli, have both stable attributes (e.g., gender of faces) and changeable attributes (e.g., emotional expression of faces). Serial dependence effects have recently been observed in various aspects of stable attributes of faces, such as facial identity (Liberman et al., 2014), facial gender (Taubert et al., 2016a), and facial attractiveness (Taubert et al., 2016b; Xia et al., 2016; Kok et al., 2017). However, inconsistent results have been found for a changeable attribute of faces – facial emotional expression. Using a sequence of morphed facial stimuli created in the two dimensions of gender (a stable attribute) and expression (a changeable attribute), Taubert et al. (2016a) presented a string of images to participants sequentially (the method of constant stimulus), and instructed them to perform two responses: the face is female or male and happy or sad. The results showed the opposite effects for the perception of these two attributes: clear positive serial dependencies for gender and well-known negative aftereffects for expression. They reasoned that there was no positive serial dependence of changeable attributes like facial expressions. However, using the method of adjustment (MOA) task where participants were instructed to adjust a variable stimulus to match a target stimulus, Liberman et al. (2018) most recently found that perception of a facial expression was biased toward recently viewed facial expressions (i.e., a positive serial dependence effect). Given these inconsistent findings, our first aim (Experiment 1) was to replicate Liberman et al. (2018)’s experiments using a two-interval forced choice (2IFC) task rather than their original MOA task, in order to investigate whether serial dependence of perceived facial expression could occur. In a 2IFC task, participants are sequentially presented with two alternative stimuli (e.g., faces) in two intervals, and are instructed to indicate which of the two stimuli is perceived to be the correct option (e.g., happier). Compared with the MOA task, the 2IFC task contributes to reduce response time and avoid interference from adjustment stimuli during the response stage (Liberman et al., 2014).

Our second aim (Experiment 2 and 3) was to further investigate the nature of the serial dependence effect of perceived facial expression. Recent work has demonstrated that serial dependence effect is associated with working memory (Papadimitriou et al., 2015; Bliss et al., 2017; Fritsche et al., 2017; Papadimitriou and Snyder, 2017). For example, the serial dependence effect on orientation perception was found to be stronger when the current visual stimulus was retained in working memory for a longer duration in an adjustment task, suggesting that working memory may be a source of this effect (Fritsche, et al., 2017). These findings were in line with previous reports that residual memory for stimuli on previous trials had influence on the judgment of the subsequent trial’s stimuli (Nelson, 1985; Visscher et al., 2009; Huang and Sekuler, 2010). However, whether working memory representations would modulate serial dependence effects of perceived facial expression with a changeable attribute has not been investigated yet.

Collectively, the current study aimed to investigate the serial dependence of perceived emotional expression and its nature. Our results in Experiment 1 showed the existence of the serial dependence of perceived emotional expression. We further found that the serial dependence effect occurred only when the first facial expression was retained in working memory for a longer duration (Experiment 2), and the serial dependence effect became weaker when a longer delay between decision and response was performed (Experiment 3), indicating that working memory maintenance modulated serial dependence of perceived emotional expression.

Experiment 1: Examining the Serial Dependence of Perceived Emotional Expression Using the Method of Constant Stimuli

In Experiment 1, we aimed to replicate the result of Liberman et al. (2018) experiments where perception of a facial expression was biased toward recently viewed facial expressions (i.e., serial dependence effect of perceived emotional expression), using the method of constant stimuli and a 2IFC task rather than their original MOA task. To verify the reliability of research results, we recruited two groups of participants (Experiment 1a and 1b) separately. They performed the same tasks but did them using different response instructions (see Procedure for details).

Methods

Participants

Sixteen participants (ten females; mean age = 21.2 years, SD = 1.9; 19–25 years) participated in Experiment 1a, and an additional sixteen participants (nine females; mean age = 20.8 years, SD = 1.4; 19–24 years) participated in Experiment 1b. All participants were naïve to the experiments, except one of the authors (S.C.) who participated in Experiment 1a. All participants of the current study had normal or corrected-normal vision, provided written informed consent and were paid after completing the experiments. All experimental procedures were approved by the School of Psychology Ethics Committee at Guizhou Normal University and conformed to the Declaration of Helsinki.

Stimuli and Apparatus

We used FaceGen Modeller 3.4¹ to generate an Asian male face model which is based on pictures of laser-scanned 3D human faces. FaceGen can generate realistic facial expressions by controlling action units parametrically, based on the Facial Action Coding System. This software has been used to morph various intensities of emotional expressions in previous studies (e.g., Huang et al., 2012; Thoma et al., 2013). In the current study, the “Expression” module of the Modeller was used to morph the FaceGen model with a neutral expression into emotional faces. Using the “Sad Expression” sub-module, we morphed sad expressions with the neutral expression, resulting in a set of 25 sad morphs with various intensities ranging from 4 to 100% sad with 4% increments. Similarly, using the “Happy Expression” sub-module, we morphed happy expressions with the neutral expression, resulting in a set of 25 happy morphs with various intensities ranging from 4 to 100% happy with 4% increments. Thus, a continuum with 50 morphed faces from sad to happy was formed. These morphed faces were coded from number 1 to 50 along the continuum (see Figure 1). All morphed faces were then cropped into oval shapes, using PhotoShop software and presented on a screen with a gray background. Note that the morphing method used in the current study was different than that used by Liberman et al. (2018) in which they generated morphs between two different expressions (e.g., from happy to sad). Morphing sad/happy with neutral could more closely produce natural expressions, compared with morphing between different expressions (Janek and Martin, 2015). The morphing method in the current study was used in previous studies (e.g., Kamachi et al., 2001; Bui et al., 2015; D’Hondt et al., 2015; Jackson and Arlegui-prieto, 2016; Nakajima et al., 2017; Orrie et al., 2018).

FIGURE 1

Figure 1. Examples of morphed emotional expressions used in Experiment 1, Experiment 2, and Experiment 3. Fifty morphs with various intensities from 100% sad to 100% happy were created using an Asian male model displaying a neutral expression. For convenience, we coded these morphs from number 1 to 50 along the continuum.

Face stimuli (6.1° × 7.9°) were presented on the center of a gamma-corrected 21-in. CRT monitor with a resolution of 1024 × 768 pixels and a refresh rate of 85 Hz. All experiments were programmed using the PsychToolbox-3 (Brainard, 1997; Pelli, 1997) running in MATLAB (MathWorks, Natick, MA, United States). Participants were required to view face stimuli at a distance of 56 cm in a dark room, and their heads were rested on a chinrest. The same facial expression stimuli and apparatus were used for all experiments.

Procedure

The method of constant stimuli and the 2IFC task was used. In each trial (see Figure 2), two facial expressions were presented one after another. The first facial expression was presented for 1000 ms, followed by a 1000 ms noise mask with a random level of intensity in grayscale for each pixel and then a 250 ms fixation cross. Next, the second facial expression was presented for 500 ms, and after a 1000 ms noise mask, the question “which facial expression was more sad” (Experiment 1a) or “which facial expression was more happy” (Experiment 1b) appeared until participants made a response by pressing “1” or “2” on the keypad. The next trial started after a 1500 ms intertrial interval. The participants were allowed to guess if necessary.

FIGURE 2

Figure 2. The schematic description of the trial sequences for the two-interval forced choice (2IFC) task and trial types in Experiment 1. The sad-previous trial (A) and happy-previous trial (B) were displayed separately. In each trial, the first facial expression was presented for 1000 ms, followed by a 1000 ms noise mask and a 250 ms fixation cross. Then the second facial expression was presented for 500 ms, and after a 1000 ms noise mask, the question “which facial expression was more sad” (Experiment 1a) or “which facial expression was more happy” (Experiment 1b) appeared until participants made a response by pressing “1” or “2” on the keypad. The next trial started after a 1500 ms intertrial interval. We classified the trial types based upon the position of the n-back first emotional expression in the morph continuum (see Analysis for details).

In regard to different exposure durations of the first and the second facial expressions, we followed the trial sequence for the 2IFC task in Experiment 2 of Liberman et al. (2014), although they investigated the serial dependence effect in the perception of face identity rather than emotional expressions. The same rationale and parameters for exposure durations as that in experiment 2 of Liberman et al. (2014) were used. We hypothesized that the one-back first facial expression had a stronger pull to the next trial’s first facial expression than the one-back second facial expression, given that the exposure duration of the first facial expression (i.e., 1000 ms) doubles as that of the second facial expression (i.e., 500 ms). If equal exposure durations for the first and the second facial expressions were used in a 2IFC task, the one-back serial dependence effect could not be found. This is because the first and the second facial expressions may have different emotional dimensions (i.e., sad versus happy), thereby producing opposite equal pulls to the next trial’s first facial expression. These opposite pulls may cancel each other out, leading to no serial dependence effect.

Following the previous work (Liberman et al., 2014), we randomly selected the first facial expression from 26 morphed expressions ranging from number 13 to 38 in the center of the morphed continuum in each trial. The second facial expression could have differences of ±12, ±6, or 0 bins from the first facial expression. For example, the facial expression numbered 1 should be selected as the second facial expression toward the -12 bin when a facial expression numbered 13 was selected as the first facial expression. The second facial expression displayed a more sad expression (or less happy) than the first facial expression when -12 and -6 bins were used; the second facial expression displayed less sad (or a more happy expression) than the first facial expression when +12 and +6 bins were used. The same first and second facial expressions were used in the 0 bin.

Because 26 morphed expressions for the first facial expression and five types of bins for the second facial expression were available in each trial, 130 trials were included in each session. Each participant finished three sessions (i.e., 390 trials) in total and had a break for at least 10 min between two successive sessions. Each session took approximately 13 min. Participants completed 20 practice trials in which feedback was provided but not in the formal experiments.

Analysis

According to previous studies (Liberman et al., 2014; Taubert et al., 2016a) and our current design, we classified trials into two subgroups: “sad-previous” trials and “happy-previous” trials. The former referred to those trials for which the one-back first facial expression belonged to the part including sad morphs (i.e., number 13 to 25) in the continuum; the latter referred to those trials for which the one-back first facial expression belonged to the part including happy morphs (i.e., number 26 to 38) in the continuum. Participants finished equal numbers of “sad-previous” trials and “happy-previous” trials. These two types of trials were randomly presented. For each participant, psychometric functions of the “sad-previous” trials and the “happy-previous” trials were separately fitted with a logistic equation as follows:

\begin{matrix} P = \frac{1}{1 + e^{- a (x - b)}} \end{matrix}

Where x represents five bins (i.e., ± 12, ± 6 and 0), P represents percent of first faces chosen as more sad or happy on each bin, the parameters a and b are the slope and the point of subjective equality (PSE) for the logistic function, respectively. For each participant, the difference of the PSE values (ΔPSE) between the “sad-previous” trials and the “happy-previous” trials was calculated. This difference would be zero if serial dependence or visual adaptation failed to occur. Using one-sample t-test, we evaluated statistical significance between PSE differences and zero. For the t-tests, the Cohen’s d was used as an indicator for the effect size (Cohen, 1992).

Results

In Experiment 1a, a significant leftward shift of the one-back sad-previous curve relative to the 1-back happy-previous curve (i.e., significant ΔPSE) was found [t(15) = 2.55, p = 0.022, d = 0.64] (Figure 3A); similarly, in Experiment 1b a significant leftward shift of the one-back happy-previous curve relative to the one-back sad-previous was also found [t(15) = 2.98, p = 0.009, d = 0.75] (Figure 3D). These results suggested that perception of the current trial’s first facial expression was pulled toward one-back trial’s first facial expression (one-back serial dependence effect).

FIGURE 3

Figure 3. Grand average psychometric functions for Experiment 1a (top panel) and 1b (bottom panel). The horizontal coordinate represents the differences between the first and second facial expressions in the morph continuum. Trials in the –6 and –12 bin denote that the second facial expression displayed more sad (or less happy) than the first facial expression; trials in the 6 and 12 bin denote that the second facial expression displayed less sad (or more happy) than the first facial expression; trials in the 0 bin denote the same first and second facial expressions. Note that, for comparison, Experiment 1a and 1b have an opposite direction on the horizontal coordinate values. The vertical coordinate represents percent of the first facial expression chosen as more sad (Experiment 1a) or more happy (Experiment 1b) on the corresponding five bins. The red and blue curves represent the fitting curves for the “sad-previous” and “happy-previous” trials, respectively. ΔPSE denotes the differences of in PSE values (i.e., magnitude of serial dependence effects) between the “sad-previous” trials and the “happy-previous” trials. One-back, two-back and three-back serial dependence effects are shown separately in the top panel (A–C) for Experiment 1a and in the bottom panel (D–F) for Experiment 1b. See Procedure and Analysis for more details.

There were no 2-back serial dependence effect [Experiment 1a: t(15) = 1.70, p = 0.110, d = 0.43, see Figure 3B; Experiment 1b: t(15) = 0.97, p = 0.347, d = 0.24, see Figure 3E] and 3-back serial dependence effects [Experiment 1a: t(15) = 1.03, p = 0.320, d = 0.26, see Figure 3C; Experiment 1b: t(15) = 0.32, p = 0.756, d = 0.08, see Figure 3F]. Similar results of Experiment 1a and 1b indicated that different response instructions had no impact on the serial dependence effect in the perception of perceived facial expression.

Given that average response times across participants were about 604 ± 228 ms in Experiment 1a and about 512 ± 154 ms in Experiment 1b, the one-back first facial expression appeared before the current first facial expression about 5854 ms and 5762 ms before in Experiment 1a and 1b, respectively. In addition, the fitted slopes (i.e., the parameter a) between the “sad-previous” and the “happy-previous” curves showed no significant difference [Experiment 1a: t(15) = 0.92, p = 0.374, d = 0.23; Experiment 1b: t(15) = 1.29, p = 0.216, d = 0.32], indicating that there was no difference in sensitivity between these two types of trials. Similar results were found when the author’s (S.C.) data was removed for Experiment 1a (see Supplementary Materials for details).

Experiment 2: Stronger Serial Dependence of Perceived Emotional Expression When Lengthening the Duration of the First Face Retained in Working Memory

Experiment 1 replicated and extended previous results demonstrating the existence of the serial dependence of perceived emotional expression (Liberman et al., 2018), using a different method (constant stimuli versus adjustment). Although recent works have demonstrated that working memory representations are associated with the serial dependence effects of stable attributes of visual stimuli (Papadimitriou et al., 2015; Bliss et al., 2017; Fritsche et al., 2017; Papadimitriou and Snyder, 2017), it is unclear whether the serial dependence effect of perceived emotional expression (a changeable visual attribute) would be modulated by working memory representation.

Given that the design of Experiment 1 required participants to compare the first faces with the second faces across time in the 2IFC task, a working memory component was necessarily involved. More specifically, in a 2IFC task participants need to judge how a stimulus they are looking at differs from another stimulus presented a few seconds ago, so they must draw on their memory for making the judgment. When the time interval between the two stimuli is longer, serial dependence from previous trials may impact upon their judgment more. In Experiment 2, we shortened or lengthened the duration of the first face retained in working memory in a 2IFC task to observe whether serial dependence effects would emerge.