Criterion Validity of the Implicit Positive and Negative Affect Test: Prediction of Facial Affect Perception

This study focused on the criterion-related validity of the Implicit Positive and Negative Affect Test (IPANAT). The IPANAT is thought to be a measure of automatic activation of cognitive representations of affects. In this study, it was investigated whether implicit affect scores differentially predict ratings of facial emotions over and above explicit affectivity. Ninety-six young female participants completed the IPANAT, the Positive and Negative Affect Schedule (PANAS) as an explicit measure of state and trait affectivity, and a task for the perception of facial emotions. Implicit negative affect predicted the perception of negative but not positive facial emotions, whereas implicit positive affect predicted the perception of positive but not negative facial emotions. The observed double-dissociation in the correlational pattern strongly supports the validity of the IPANAT as a measure of implicit affectivity and is indicative of the orthogonality and thus functional distinctness of the two affect dimensions of the IPANAT. Moreover, such affect-congruent correlations were absent for explicit affect scales, which additionally supports the incremental validity of the IPANAT.

Introduction

For a long time, psychological research has been struggling with measuring affects without tapping into problems associated with explicit assessment. Explicit or self-report measures, which assess affectivity directly, are biased by factors, such as social desirability, impression management strategies, and reduced introspective ability (Mitchell and Tetlock, 2015). To avoid such difficulties and to make other facets of the construct accessible, Quirin et al. (2009a) developed the Implicit Positive and Negative Affect Test (IPANAT). Explicit affect is formed by conscious reflections and refers to conscious affective experiences (Quirin and Bode, 2014).

The IPANAT was designed to measure implicit affect, defined as the automatic activation of cognitive representations of affective experiences. In this test, participants are instructed to provide ratings concerning the degree to which artificial words from a putative artificial language express a number of affective states or traits (Quirin et al., 2018). The IPANAT makes use of the principle of affect infusion according to which affect exerts an impact on evaluative processes influencing affective perceptions of unrelated objects (Forgas, 2001). In-line with two-dimensional models of affect (Watson et al., 1988), positive affect and negative affect are treated as two different dimensions in the IPANAT. The IPANAT scales have shown relationships of small-to-moderate strength with explicit measures of the same affect type (Quirin et al., 2009a). In study 2 of Quirin et al. (2009a), the IPANAT scales correlated with tests and tasks assessing aspects of implicit affectivity, the Neutral Objects Satisfaction Questionnaire that requires ratings on satisfaction with everyday objects (Judge and Bretz, 1993), and a memory retrieval task of affective contents based on completion of word fragments (Graf et al., 1982). These findings provide important indications of the validity of the IPANAT as a measure of implicit affect.

In this study, aspects of criterion-related validity were investigated by correlating the IPANAT scales with judgments of facial emotions perceived. Criterion-related validity refers to the degree to which a measurement can predict an outcome for another measure (Cohen and Swerdlik, 2005). A test has criterion validity if it is useful for accurately predicting performance or behavior in another (past, present, or future) situation. We used schematic faces with different expressions to assess the perception of emotions in faces (Bouhuys et al., 1997). The German version of the Positive and Negative Affect Schedule (PANAS; Krohne et al., 1996), which is a self-report questionnaire, was administered to assess explicit positive and negative state and trait affects (i.e., actual, persistent, and habitual affects). Based on previous findings concerning the implicit processing of lexical material (see Quirin et al., 2009a, study 2), it was hypothesized that implicit but not explicit affect scores (i.e., positive, and negative state and trait affect as assessed by the PANAS) predict judgments of facial emotions. Specifically, we expected that implicit positive affect is associated with enhanced attribution of positive affect, and implicit negative affect is associated with heightened attribution of negative affect, regardless of the type of affective expression. We also examined the relationships of IPANAT scales with explicit state and trait affect as assessed by the PANAS.

Materials and Methods

Participants

Participants were recruited using social media and announcements in buildings of the university. Most of the participants were university students, and all of them were women. Exclusion criteria were a lifetime history of psychiatric or neurological diseases, psychotropic medication use, and no native German speaking. The present sample consisted of 96 women with a mean age of 23.9 years (SD = 3; range: 19–30). School education of the participants had a mean duration of 12.3 years (SD = 0.6), and the mean verbal IQ as assessed by the Mehrfachwahl-Wortschatz-Intelligenz test (MWT-B) was 112.1 (SD = 10.7). Our study was carried out in accordance with the code of ethics (Declaration of Helsinki) of the World Medical Association (2013). Our study was approved by the ethics committee at the University of Leipzig, Medical Faculty (reference number 138-14-14042014). All participants gave written informed consent after reading a description of the study and received financial compensation.

Questionnaires and Tests

Implicit positive and negative affects were measured by the IPANAT (Quirin et al., 2009a). The IPANAT assesses affect in an indirect way by asking participants to rate the extent to which artificial words (e.g., VIKES) from a putative artificial language express a certain mood on a four-point scale. Each of the artificial words is presented along with three positive and three negative affective words. For assessing explicit positive and negative state and trait affects, the German version of the PANAS (Krohne et al., 1996) was used. The PANAS consists of 20 adjectives, 10 describing positive and the other 10 negative affective states. On a five-point scale, the participants rated how much they feel like this at the moment (state version) or in general (trait version). Verbal intelligence was assessed by the MWT-B (Lehrl, 2005).

Perception of Facial Emotions

The perception of facial expressions (PFE) task consisted of 12 schematic oval faces as applied by Bouhuys et al. (1997). The line drawings were composed of one type of eyes and nose, four eyebrows, and three mouth types (see Bouhuys et al., 1997, for stimulus material). The schematic faces had negative, positive, neutral, and ambiguous expressions. Images of faces remained on the screen until a response was given. To control stimulus presentation and record performance, Presentation software (Version 16.3, Neurobehavioral Systems, Berkeley, CA, USA; www.neurobs.com) was used. Participants were instructed to evaluate schematic drawings of faces with respect to the degree to which they express certain affects. At the top of the screen, a question (e.g., “How strongly does the face express sadness?”) was presented. At the bottom of the screen, the response format was shown. Responses were given on a five-point scale ranging from 0% (not at all, 1), 25% (very little, 2), 50% (little, 3), and 75% (quite strongly, 4) to 100% (very strongly, 5), while the randomly selected facial stimuli appeared in the center of the screen. Following Bouhuys et al. (1997), subjects had to evaluate with respect to seven basic or relational affects [positive: happiness (Freude), attraction (Anziehung); negative: fear (Angst), anger (Ärger), sadness (Traurigkeit), disgust (Ekel), and rejection (Zurückweisung) (see also Hale et al., 1998)]. In total, participants made 84 judgments in the PFE task (12 faces × 7 affect ratings). Answers were given on a keyboard by pressing the “1”, “2”, “3”, “4”, or “5” key. In Figure 1, we show an example of a trial of the PFE task. Two cumulative scores were calculated from the evaluative PFE data: (1) Judgments of the 5 negative categories were averaged over the 12 facial stimuli, and (2) judgments of the 2 positive categories were averaged over the 12 facial stimuli.

FIGURE 1

Figure 1. In the perception of facial expressions (PFE) task, schematic faces were displayed. In the present example, a face with a neutral expression is shown. Images of faces remained on the screen until response. Participants had to evaluate the face with respect to the degree to which it expresses a certain affect (in the example the question is “How strongly does the face express fear?”). At the bottom of the screen, the response format was shown. Participants evaluated the expression of seven affects (positive: happiness, attraction; negative: fear, anger, sadness, disgust, and rejection). Answers were given on a keyboard by pressing the “1”, “2”, “3”, “4”, or “5” key.

General Procedure

Testing sessions were conducted individually in a quiet room free from auditory and visual distractions. Tests were administered in a fixed order with participants starting to complete the IPANAT and the PANAS state scales and proceeding with the PFE task. A laptop (Dell Latitude E6510) was used for the presentation of the PFE task. Thereafter, participants filled out the PANAS trait scales and the MWT-B.

Statistical Analysis

Product moment correlation analyses were performed to explore the relationships between measures of affectivity, intelligence, and PFE. In addition, hierarchical regression analyses were conducted for perceived positive affect and perceived negative affect. In the first step, explicit positive and explicit negative affects (state and trait) as assessed by the PANAS and intelligence (IQ scores) were entered as predictors. The second step included entering implicit positive affect and implicit negative affect as measured by the IPANAT. Results were considered significant at p < 0.05, two-tailed. All calculations were administered using SPSS 25.0 (IBM Corp., Armonk, NY, USA).

Results

As expected, implicit positive affect (IPANAT-PA) showed a positive correlation with the PANAS-PA state, whereas implicit negative affect (IPANAT-NA) was not correlated with any of the PANAS scores (see Table 1). Moreover, the PANAS-PA state showed a correlation with the PANAS-PA trait. PANAS-NA state was correlated with the PANAS-NA trait. Negative correlations of PANAS-NA state were observed with PANAS-PA state and PANAS-PA trait. Intelligence was correlated only with the PANAS-PA trait. IPANAT-PA correlated positively with the perception of positive affect in facial expressions. In addition, a positive correlation between the PANAS-NA trait and the perception of positive affect was revealed. Finally, IPANAT-NA correlated positively with the perception of negative affect (see Table 1).

TABLE 1

Table 1. Descriptive statistics and correlations between psychological measures.

Two hierarchical regression analyses were conducted (for perceived positive affect and perceived negative affect as dependent variables). The regression model concerning the perception of positive affect yielded in the first step (which included PANAS-PA state and trait, PANAS-NA state and trait, and verbal IQ as measured by the Mehrfachwahl-Wortschatz-Intelligenz test [MWT-B]) a significant result: Explicit negative trait affect was a significant predictor of positive affect perception (see Table 2). In the second step, after the inclusion of IPANAT-PA and -NA, implicit positive affect was found to significantly predict perceived positive affect in facial expressions (see Table 2). The first step of the regression analysis regarding the perception of negative affect did not show a significant result. In the second step, after the inclusion of IPANAT-PA and -NA, implicit negative affect predicted the perception of negative emotions in faces (see Table 2).

TABLE 2

Table 2. Hierarchical multiple regression analyses with perceived affect in the PFE task as dependent variables and affect measures and MWT-B as predictors.

Discussion

For this study, we expected that implicit but not explicit affect predicts ratings of facial emotions. Supporting this hypothesis, we found that implicit negative affect was the only variable predicting the perception of negative affect in schematic faces. Moreover, IPANAT-PA predicted the attribution of positive affect to faces, independent of explicit affect. The observed correlations with face perceptions support the criterion-related validity of the IPANAT as a measure of implicit affectivity complementing and extending previous work based on the processing of lexical material (Quirin et al., 2009a, study 2). Importantly, in our investigation, participants were not asked to describe the way the facial expressions make them feel but they had to judge how strongly the faces express positive or negative emotions. In study 2 of Quirin et al. (2009a), it was shown that implicit affect as assessed by the IPANAT is related to the degree to which positive and negative contents or experiences are retrieved from memory. These findings are in-line with the conceptualization of implicit affect as automatic activation of affect representations, which influences memory retrieval and judgmental processes. The correlation results obtained reveal incremental predictive validity of the IPANAT suggesting that this measure can provide important information on how people assess and interpret social signals, above and beyond explicit affect measures. Finally, the present data show additional validity for the functional distinctness of the two affect dimensions of the IPANAT.

In our study, an unexpected correlation was found between explicit negative trait affect and perceived positive affect. This could mean that heightened self-reported habitual negative affect goes along with an increased perception of positive affect in facial expressions. This correlation might result from mood repair efforts in our healthy subjects. There is evidence from eye-tracking studies that healthy individuals increase their attention to positive images and happy faces to counteract and repair sad or negative moods (Sanchez et al., 2014; Newman and Sears, 2015). Possibly, a positive bias in the affective perception of the facial expressions of others may in the long run contribute to a reduction in negative affect experienced in everyday life.

Evidence for criterion and construct validity of the IPANAT has also been provided by recent neuropsychological and neurobiological studies. It was shown that implicit affect as assessed by the IPANAT predicts attention bias to negative faces when looking at crowds of faces (Bodenschatz et al., 2018), recognition of briefly presented angry body posture, and neural activation of subcortical structures in response to threatening body expressions (Suslow et al., 2015). Moreover, implicit affect as measured by the IPANAT has been found to be a predictor of autonomous nervous system reactions, such as stress-related cardiovascular activity (van der Ploeg et al., 2016) and cortisol secretion (Quirin et al., 2009b), beyond explicit methods of affect assessment (see Weil et al., 2019, for a review). Proof for cross-cultural validity of the IPANAT emerged recently (Sulejmanov and Spasovski, 2017; Quirin et al., 2018; Hernández et al., 2020b). It should be noted that a short form of the IPANAT consisting only of 18 items, the IPANAT-18, has been developed. The IPANAT-18 showed satisfactory psychometric properties and seemed to represent a promising tool for economically measuring implicit affectivity (Hernández et al., 2020a).

In our study, we also examined the correlations of the IPANAT scales with explicit state and trait affect measures. According to Quirin et al. (2009a), the IPANAT scales show relationships of small-to-moderate strength with explicit measures of the same affect type. Strengthening to some extent the concurrent validity of the IPANAT, we found a significant correlation between implicit positive affect and positive state affect as measured by the PANAS in our sample. The correlations we observed between IPANAT-PA and PANAS-PA (state and trait) are within the range of correlation coefficients reported by Quirin et al. (2009a, 2018). In our investigation, the IPANAT-NA did not correlate significantly with explicit measures of negative affect. The correlations found in our study between IPANAT-NA and PANAS-NA (state and trait) of 0.1 are low and can be interpreted as indicating a small effect size. However, they seem not to differ substantially from correlations observed in the previous research studies. In rather large samples from seven countries (Austria, China, Italy, Poland, Russia, USA, and Uzbekistan), correlations between IPANAT-NA and PANAS-NA (trait) varied between 0.16 and 0.28 (with a median r of 0.21; Quirin et al., 2018). Thus, the strength of the relationship between implicit and explicit negative affect appears to be in general rather weak (especially, when implicit affect is measured only once, and no previous affect induction is carried out).

Some limitations of our study have to be acknowledged. The number of included subjects is relatively low. Additionally, our sample consisting of young, well-educated women is not representative of the general population. This limits the generalizability of our results. Against this background, it seems necessary to replicate our findings in populations other than female college students. In future studies on the criterion validity of the IPANAT, it could be useful to administer real faces instead of schematic line drawings and examine whether implicit affect as assessed by the IPANAT also predicts perceived affect in these facial stimuli. The present investigation follows a traditional approach of test validation. According to this approach, validation is a process in which a variety of different forms of validity (such as criterion validity) are examined, and the test is set into a network of relations to outside criteria (Messick, 1989). Alternative modern forms of validation refer to Item Response Theory (IRT) models that link responses to items with underlying latent constructs through formalized statistical modeling (Geiser and Eid, 2010). IRT procedures, such as the Graded Response Model, have shown to improve positive and negative affect estimation (Zanon et al., 2016). IRT models have proven to be useful tools not only in the development of scales but also in the testing of theoretical assumptions (Lang and Tay, 2021). Future research studies on the convergent and discriminant validity of the IPANAT might benefit from administering latent multi trait-multimethod IRT models (see Runge and Lang, 2019, for an example of use in research on self-evaluation of implicit motives).

In sum, the present results support the validity of the IPANAT as a measure of implicit affect by showing differential relationships of the implicit positive and negative affect scales with perceptions of affect in facial expressions.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethikkommission, Medizinische Fakultät, Universität Leipzig. The participants provided their written informed consent to participate in this study.

Author Contributions

VG, AK, and TS conceived and designed the experiment. VG collected the data. A-SW, MQ, TS, FS, and AK outlined the manuscript. A-SW, VG, and TS analyzed the data. A-SW wrote the manuscript with revisions and contributions from TS, VG, FS, AK, and MQ.

Funding

This work was partially made possible through a grant from the Templeton Rlg. Trust (TRT 0119) supporting MQ.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Bodenschatz, C. M., Skopinceva, M., Kersting, A., Quirin, M., and Suslow, T. (2018). Implicit negative affect predicts attention to sad faces beyond self-reported depressive symptoms in healthy individuals: an eye-tracking study. Psychiatry Res. 265, 48–54. doi: 10.1016/j.psychres.2018.04.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Bouhuys, A. L., Geerts, E., and Mersch, P. P. A. (1997). Relationship between perception of facial emotions and anxiety in clinical depression: does anxiety-related perception predict persistence of depression? J. Affect. Disord. 43, 213–223. doi: 10.1016/S0165-0327(97)01432-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Cohen, R. J., and Swerdlik, M. E. (2005). Psychological Testing and Assessment: An Introduction to Tests and Measurement, (6th ed.). New York, NY: McGraw-Hill.

Google Scholar

Forgas, J. P. (2001). “The Affect Infusion Model (AIM): An integrative theory of mood effects on cognition and judgments,” in Theories of Mood and Cognition: A User's Guidebook, ed L. L. Martin and G. L. Clore (New York, NY: Psychology Press), 99–134.

Google Scholar

Geiser, C., and Eid, M. (2010). “Item-Response-Theorie,” in Handbuch der sozialwissenschaftlichen Datenanalyse, ed C. Wolf and H. Best (Wiesbaden: VS Verlag für Sozialwissenschaften), 311–332. doi: 10.1007/978-3-531-92038-2_14

CrossRef Full Text | Google Scholar

Graf, P., Mandler, G., and Haden, P. E. (1982). Simulating amnesic symptoms in normal subjects. Science 218, 1243–1244. doi: 10.1126/science.7146909

PubMed Abstract | CrossRef Full Text | Google Scholar

Hale, W. W., Jansen, J. H. C., Bouhuys, A. L., and van den Hoofdakker, R. H. (1998). The judgment of facial expressions by depressed patients, their partners and controls. J. Affect. Disord. 47, 63–70. doi: 10.1016/S0165-0327(97)00112-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Hernández, G. P., Edo, S., Quirin, M., and Rovira, T. (2020a). A brief version of the Implicit Positive and Negative Affect Test (IPANAT-18). Psychol. Belg. 60, 315–327. doi: 10.5334/pb.544

PubMed Abstract | CrossRef Full Text | Google Scholar

Hernández, G. P., Rovira, T., Quirin, M., and Edo, S. (2020b). A Spanish adaptation of the implicit positive and negative affect test (IPANAT). Psicothema 32, 268–274. doi: 10.1037/t78010-000

PubMed Abstract | CrossRef Full Text | Google Scholar

Judge, T. A., and Bretz, R. D. (1993). Report on an alternative measure of affective disposition. Educ. Psychol. Meas. 53, 1095–1104. doi: 10.1177/0013164493053004022

CrossRef Full Text | Google Scholar

Krohne, H. W., Egloff, B., Kohlmann, C.-W., and Tausch, A. (1996). Untersuchungen mit einer deutschen Version der “Positive and Negative Affect Schedule” (PANAS). Diagnostica 42, 139–156. doi: 10.1037/t49650-000

CrossRef Full Text | Google Scholar

Lang, J. W. B., and Tay, L. (2021). The science and practice of item response theory in organizations. Ann. Rev. Organ. Psychol. Organ. Behav. 8, 311–338. doi: 10.1146/annurev-orgpsych-012420-061705

PubMed Abstract | CrossRef Full Text | Google Scholar

Lehrl, S. (2005). Mehrfachwahl-Wortschatz-Intelligenztest: MWT-B. Balingen: Spitta.

Messick, S. (1989). Meaning and values in test validation: the science and ethics of assessment. Educ. Res. 18, 5–11. doi: 10.3102/0013189X018002005

CrossRef Full Text | Google Scholar

Mitchell, G., and Tetlock, P. E. (2015). “Implicit attitude measures,” in Emerging Trends in the Social and Behavioral Sciences, ed R. Scott and S. Kosslyn (New York, NY: John Wiley), 1–14.

Google Scholar

Newman, K. R., and Sears, C. R. (2015). Eye gaze tracking reveals different effects of a sad mood induction on the attention of previously depressed and never depressed women. Cogn. Ther. Res. 39, 292–306. doi: 10.1007/s10608-014-9669-x

CrossRef Full Text | Google Scholar

Quirin, M., and Bode, R. C. (2014). An alternative to self-reports of trait and state affect: the implicit positive and negative affect test (IPANAT). Eur. J. Psychol. Assess. 30, 231–237. doi: 10.1027/1015-5759/a000190

CrossRef Full Text | Google Scholar

Quirin, M., Kazén, M., and Kuhl, J. (2009a). When nonsense sounds happy or helpless: the implicit positive and negative affect test (IPANAT). J. Pers. Soc. Psychol. 97, 500–516. doi: 10.1037/a0016063

PubMed Abstract | CrossRef Full Text | Google Scholar

Quirin, M., Kazén, M., Rohrmann, S., and Kuhl, J. (2009b). Implicit but not explicit affectivity predicts circadian and reactive cortisol: using the implicit positive and negative affect test. J. Pers. 77, 401–425. doi: 10.1111/j.1467-6494.2008.00552.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Quirin, M., Wróbel, M., Norcini Pala, A., Stieger, S., Brosschot, J., Kazén, M., et al. (2018). A Cross-Cultural Validation of the Implicit Positive and Negative Affect Test (IPANAT). Eur. J. Psychol. Assess. 34, 52–63. doi: 10.1027/1015-5759/a000315

CrossRef Full Text | Google Scholar

Runge, J. M., and Lang, J. W. B. (2019). Can people recognize their implicit thoughts? the motive self-categorization test. Psychol. Assess. 31, 939–951. doi: 10.1037/pas0000720

PubMed Abstract | CrossRef Full Text | Google Scholar

Sanchez, A., Vazquez, C., Gomez, D., and Joormann, J. (2014). Gaze fixation to happy faces predicts mood repair after a negative mood induction. Emotion 14, 85–94. doi: 10.1037/a0034500

PubMed Abstract | CrossRef Full Text | Google Scholar

Sulejmanov, F., and Spasovski, O. (2017). Psychometric properties of the macedonian version of the implicit positive and negative affect test (IPANAT–M). Rev. Psychol. 24, 39–44.

Google Scholar

Suslow, T., Ihme, K., Quirin, M., Lichev, V., Rosenberg, N., Bauer, J., et al. (2015). Implicit affectivity and rapid processing of affective body language: an fMRI study. Scand. J. Psychol. 56, 545–552. doi: 10.1111/sjop.12227

PubMed Abstract | CrossRef Full Text | Google Scholar

van der Ploeg, M. M., Brosschot, J. F., Thayer, J. F., and Verkuil, B. (2016). The Implicit positive and negative affect test: validity and relationship with cardiovascular stress-responses. Front. Psychol. 7:425. doi: 10.3389/fpsyg.2016.00425

PubMed Abstract | CrossRef Full Text | Google Scholar

Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Pers. Soc. Psychol. 54, 1063–1070. doi: 10.1037/0022-3514.54.6.1063

PubMed Abstract | CrossRef Full Text | Google Scholar

Weil, A.-S., Hernández, G. P., Suslow, T., and Quirin, M. (2019). Implicit affect and autonomous nervous system reactions: a review of research using the implicit positive and negative affect test. Front. Psychol. 10:634. doi: 10.3389/fpsyg.2019.01634

PubMed Abstract | CrossRef Full Text | Google Scholar

World Medical Association (2013). World Medical Association declaration of helsinki. ethical principles for medical research involving human subjects. J. Am. Med. Assoc. 310, 2191–2194. doi: 10.1001/jama.2013.281053

PubMed Abstract | CrossRef Full Text | Google Scholar

Zanon, C., Hutz, C. S., Yoo, H., and Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia Reflexao Critica 29:18. doi: 10.1186/s41155-016-0040-x

CrossRef Full Text | Google Scholar