Effectiveness of the “What’s Up!” Intervention to Reduce Stigma and Psychometric Properties of the Youth Program Questionnaire (YPQ): Results from a Cluster Non-randomized Controlled Trial Conducted in Catalan High Schools

Mental disorders are highly prevalent in the general population, and people who experience them are frequently stigmatized. Stigma has a very negative impact on social, academic/professional, and personal life. Considering the high rates of mental disorders among children and adolescents (13.4%) and how critical this age is in the formation of nuclear beliefs, many campaigns to combat stigma have been developed in the last decade, with mixed results. The OBERTAMENT initiative has produced various anti-stigma campaigns in Catalonia (Spain). In the present study, the main objective was to report on the effectiveness of the OBERTAMENT “What’s up!” intervention, a curricular intervention including education and social contact conducted by the teachers in the classroom with teenagers aged between 14 and 18. Prior to this, we examined the psychometric properties of the Youth Program Questionnaire (YPQ), our main outcome measure, in terms of dimensionality, reliability, and validity. A cluster non-randomized controlled trial was conducted to assess this intervention, which was tested in nine high schools situated in the Barcelona region. A convenience sample of 261 students formed the intervention group and 132 the control group (52% women, mean age = 14, SD = 0.47). The assignment to study conditions was conducted by Departament d’Ensenyament (Department of Education), Generalitat de Catalunya (Catalan Government). Participants were evaluated at baseline, post-intervention, and 9-month follow-up. The main outcome measure of this study was the YPQ. The Reported and Intended Behavior Scale (RIBS) was used as secondary outcome measure. The statistical analysis indicated that the YPQ possesses a two-factor structure (stereotypical attitudes and intended behavior) and sound psychometric properties. The multilevel mixed-effects models revealed statistically significant interactions for both study measures and post hoc intragroup analyses revealed a significant but small improvement in the YPQ and RIBS scores in the intervention group. Overall, our results indicate that “What’s up!” produced statistically significant, albeit small improvements in stereotypical attributions and intended behavior toward people with mental disorders. Some methodological limitations and the relatively low levels of stigma observed in our sample may undermine our results. The implications of our results are discussed in relation to stigma research.

Introduction

Mental disorders have a high prevalence in the general population. According to the most recent evidence, 38.2% of the general European population experiences at least one mental disorder, which corresponds to an estimated 164.7 million people (Wittchen et al., 2011). In Spain, the European Study of the Epidemiology of Mental Disorders (ESEMeD; Haro et al., 2003) concluded that approximately one out of every five people who were surveyed had presented a mental disorder at some point of their life, and 8.5% of the population had experienced a mental disorder in the previous year. Moreover, children and teenagers are a high-risk population since a worldwide prevalence of 13.4% for any mental disorder has been recently reported (Polanczyk et al., 2015).

It is noteworthy, that people experiencing mental disorders frequently deal with stigma associated with their condition (Hugo et al., 2003; Ando et al., 2013; Loch et al., 2014). Stigma is a multidimensional phenomenon that includes cognitive and behavioral aspects (Corrigan and Shapiro, 2010). The first one is, in turn, formed out of two constructs: stereotypes, which are related to knowledge (Thornicroft et al., 2007); and prejudices, which are the generalized attitudes toward members of a social group. These cognitive aspects of stigma begin to form and consolidate during adolescence (Flavell et al., 2002; Schulze et al., 2003) and their consequence is discrimination, the behavioral aspect of stigma. In the case of mental disorders, a common example would be an employer who, based on the belief that people with a mental disorder are violent (stereotype), has a negative feeling -fear, anxiety- toward them (prejudice) and, thus, decides not to hire a person who experiences one (discrimination) (Crespo et al., 2008; Ke et al., 2015).

Stigma has many consequences: it contributes to low self-esteem and quality of life (Livingston and Boyd, 2010) and has a negative effect on adequate housing, work and financial status (Rüsch et al., 2005; Sharac et al., 2010); from a clinical perspective, it has a negative influence on symptom severity and compliance with treatment (Li et al., 2014) and may trigger suicidal ideation and behavior (Rüsch et al., 2014). When stigma is internalized (i.e., self-stigma), it is associated with rejection of help, avoidance of treatment and limited prospects of recovery, among other damaging consequences (Hing and Russell, 2017). These are some reasons why stigma has been identified as one of the greatest challenges facing mental health (Hing and Russell, 2017).

In Spain, the level of stigma toward mental disorders is in line with or slightly lower than in other European countries, with mean Reported and Intended Behavior Scale (RIBS; Evans-Lacko et al., 2011) score corresponding to the 76th percentile (Aznar-Lou et al., 2016), which indicates relatively low levels of stigma. However, this does not mean that the Spanish population should not be subjects of anti-stigma interventions as, for instance, knowledge of mental disorders has been found to be low. A study conducted by López-Ibor concluded that 83% of the surveyed Spanish population knew nothing about schizophrenia and 33% did not know about the origin and causes of the disorder even though 44% affirmed that schizophrenia is not a curable illness. In other countries, such as the United Kingdom, it was reported that the general population, and youngsters particularly, had many misconceptions about mental disorders (Crisp et al., 2000). Similarly, Crespo et al. (2008) reported a moderate degree of knowledge about the treatment of mental disorders, the work options for people who experience them, and the causes and degree of awareness of the disorder in a general population sample.

Nonetheless, knowledge about mental disorders is not the only aspect to be improved regarding stigma, as there is evidence that public attitudes have not changed over the last two decades, or have even become worse regarding people with psychotic disorders (Hansson and Markström, 2014). Some studies have identified several personal characteristics that are associated with having a higher degree of stigma toward people who experience mental disorders, such as being male (Stickney et al., 2012), having had no previous contact with people with mental conditions (Batterham et al., 2013), lower educational levels and living alone (Coppens et al., 2013). In Spain, some factors such as having completed secondary or university education and having had contact with people with mental problems are related to a better attitude toward mental disorders and more favorable intended behavior (Aznar-Lou et al., 2016).

Bearing this in mind, various approaches have attempted to reduce stigma toward mental disorders through education and social contact. Corrigan and Penn (1999) considered educational interventions to be one of the most effective ways of reducing stigma. Educational approaches often challenge inaccurate stereotypes of mental disorders, replacing them with factual information (Corrigan and Shapiro, 2010; Economou et al., 2012). These interventions are frequently addressed to children and teenagers as they constitute one of the most indicated populations for many reasons: studies have proved that children do not yet have a consolidated idea of what mental illness denotes (Corrigan and Watson, 2007), and the personality traits that constitute the foundation for stereotype endorsement are not well-entrenched until adolescence (Flavell et al., 2002), considering that this stage lasts, approximately, from 13 to 18 years old. Besides, as previously mentioned, adolescents form a population which is severely affected by mental disorders, in part due to the great challenges that this life stage entails (e.g., identity definition, sexual role and body changes, academic goals).

Therefore, interventions targeted at these age groups seem to be especially useful, not only with preventive aims but also to combat the stigma that children and adolescents already begin to experience and that frequently damages their social and academic development. Some countries, such as England, have prioritized the topic of improving mental health knowledge and reducing stigma in schools and colleges. In this case, the Public Health England and the Children & Young People’s Mental Health Coalition produced the Promoting Children and Young People’s Emotional Health and Wellbeing (2015) guide, with key evidence-based guidelines for teachers and college principals to create a whole-school approach to promoting emotional health and well-being. This approach does not simply involve a workshop on mental health, but a series of regular practices to be carried out in every school space. By using this model, it may be possible to overcome the limitations of the short-duration educational interventions that have been commonly used until date, which have generally been related to small and short-term improvements in attitudes toward mental disorders in adolescents (Bulanda et al., 2014). These interventions’ effects have been reported to last over 1–6 months (Pinfold et al., 2003; Schulze et al., 2003), and regarding its content, it has been assessed that adding social contact can either imply lower effects for adolescents (Corrigan et al., 2012) or no difference at all in comparison to education alone (Schulze et al., 2003; Warner, 2005; Yamaguchi et al., 2011).

The evaluation of the effectiveness of educational interventions is generally carried out by self-reported measures that evaluate both the cognitive and behavioral components of stigma. Many different measures have been developed, but most present some methodological issues which question their adequacy, and there is no consensus on which one to use (Sakellari et al., 2011). Furthermore, validations of stigma measures oriented to teenagers are scarce worldwide (Ochoa et al., 2015). In this regard, the Youth Program Questionnaire (YPQ), an instrument developed for the campaign “Opening Minds” which was conducted in Canada in 2013 (Stuart et al., 2014), is considered to be a good instrument as it measures both the cognitive and behavioral aspects of stigma and is addressed specifically to adolescents. Nonetheless, its psychometric properties have yet to be established.

In Catalonia, the OBERTAMENT initiative, created in 2010, has developed various campaigns to fight stigma associated with mental disorders. The present study has the main objective of evaluating the effectiveness of “What’s up!” on reducing perceived stigma in a Catalan sample of adolescent students using the self-report YPQ as primary outcome and the RIBS (Evans-Lacko et al., 2011) as a secondary outcome. “What’s up!” is a curricular intervention which includes both education and social contact, created by the OBERTAMENT initiative and addressed to 14–18 year-old Catalan students to reduce stigma levels. Since the YPQ has not yet been validated, its psychometric properties in our Catalan adolescent sample are also reported.

Materials and Methods

Study Design

This is a cluster non-randomized controlled trial designed to assess the intervention “What’s up!” in nine high schools in the region of Barcelona (Spain) with pre–post and follow up evaluations.

This study was performed in accordance with ethical standards established in the 1964 Declaration of Helsinki and its subsequent updates and established in the World Psychiatric Association Declaration of Madrid. The study protocol was approved by the ethics committee at the Sant Joan de Déu Foundation (CEIC PIC-107-14; Esplugues de Llobregat, Spain). We report this non-randomized controlled trial following Transparent Reporting of Evaluations with Non-randomized Designs guidelines (TREND; Des Jarlais et al., 2004).

Participants

A total of 446 students aged between 14 and 18 years were recruited from nine different high schools: five (Pere Vives i Vich, Montbui, Salvador Claramunt, Escola Pia, and Maristes) were assigned to the intervention group, and four (Pla de les Moreres, Escola Anoia, IES Joan Mercader, and IES Guinovarda) to the control group. There were no other inclusion/exclusion criteria for participation in this study. All centers were located in the Barcelona region. These high schools were selected by convenience by the Catalan Government (Generalitat de Catalunya) Department of Education. This public entity performed the allocation to study conditions and matched high schools considering the number of students and whether they were private (Escola Pia, Maristes and Escola Anoia) or public centers (Pere Vives i Vich, Montbui, Salvador Claramunt, Pla de les Moreres, IES Joan Mercader and IES Guinovarda). All nine high schools are coeducational and have a maximum of 25 students per class.

Intervention

The “What’s up!” intervention consists of a multifocal action to combat stigma from the classroom. The teachers were given a manual with nine didactic units which contained information, examples, and exercises about mental health problems, and they were in charge of using these units in their classes, with the freedom to choose how and when to do so. This intervention model was conceived to make “What’s up!” truly implementable in the real context of high schools. It was indicated that every unit had to be implemented for 1 week. The materials that were selected for each unit were chosen by the OBERTAMENT team and based on evidence from previous effective interventions. The protocol explained in detail different types of exercises that the students should do both individually and in groups for every didactic unit. The teachers were given instructions to employ at least three of the nine didactic units, with the first and last considered to be “essential.”

The nine didactic units corresponded to different school subjects: Language (Catalan and Spanish), Foreign Language (English and French), Sciences (Biology and Geology), Mathematics, Physical Education, and Culture and Ethical Values. It was in this last unit that a person with a mental disorder reported firsthand experiences to the students in the classroom, providing the social contact component of the intervention. Table 1 shows which units were applied in each high school.

TABLE 1

TABLE 1. Units of the curricular intervention and their implementation in every high school.

Although the teachers did not receive any specific training from OBERTAMENT on how to provide support on mental health disorders, general knowledge about what mental disorders are and how to manage classrooms with heterogeneous needs are part of every teacher’s regular academic training.

Further information on this project is available here (in Spanish): https://obertament.org/es/educacion/proyecto-what-s-up.

Procedure

After an initial meeting with the education department and all the heads of the high schools, the “What’s up!” project team presented the curricular project protocol and the study goals to the teachers who would implement it.

Data from study participants were obtained at three different times: before the intervention, after the intervention (1–3 months, depending on the high school), and at follow-up 9 months from study commencement. Students were aware of their participation in a study but they did not know to which condition their high school was allocated. However, teachers administering the interventions and those assessing outcomes were not blind to study allocation. All three evaluations were conducted in the participants’ classrooms using a paper and pencil self-administered questionnaire. As they were not adults, informed consent was signed by their parents or legal guardians prior to the first evaluation. The same version of the battery of measures was administered in all high schools.

The YPQ was translated from English into Catalan by two native bilingual English/Catalan speakers. Any discrepancies between the Catalan and English versions were resolved by agreement.

Measures

To evaluate previous contact and experience with people with a mental disorder, four ad hoc questions in dichotomous format (yes/no) were included, along with a further question regarding personal experiences of having a mental disorder.

• The main outcome measure was the YPQ adapted to Catalan, an instrument developed by the Canadian initiative “Opening minds” (Stuart et al., 2014) which has two 11-item scales; the Stereotype Scale (YPQ-SS) which measures stereotypic attributions (controllability of the illness, potential for recovery, and potential for violence and unpredictability) and the Social Acceptance Scale (YPQ-SAS) which measures behavioral intentions related to social acceptance. The YPQ is rated on a five-point Likert scale (from 1 to 5) where lower scores indicate lower levels of stigma toward people with a mental disorder. Items 14, 16, 19, 20, 21, and 22 are reverse scored. This study uses the total mean and subscales scores. The psychometric characteristics of this questionnaire are described below. Two exemplary items of this questionnaire are “People with mental illnesses need to be locked away” (YPQ-SS) and “If I knew someone had a mental illness I would not date them” (YPQ-SAS). All the items can be found in the table added as Supplementary Material.

• The Catalan version of the RIBS (Evans-Lacko et al., 2011) was also used to measure intended behavior in relation to future contact with people with a mental disorder (intention to live with, work with, live nearby and continue a relationship with someone with a mental disorder). The four items are rated on a five-point Likert scale from 1 (strongly agree) to 5 (strongly disagree) so that lower scores indicate more favorable intended behavior. In the present study, we use the mean item score. In this project, internal consistency (Cronbach’s α) of RIBS was 0.81, indicating adequate reliability (α ≥ 0.8 is defined as “good”; Nunnally, 1978). An example of an item in this scale is “In the future, I would be willing to live nearby to someone with a mental health problem.”

Data Analyses

Student’s t-test for continuous values and χ² tests with continuity corrections for categorical values were computed on baseline measures and socio-demographic variables to examine differences between the two groups.

Firstly, to analyze the effectiveness of an intervention, we need outcome measures that are well-designed and have empirically proven their reliability and validity. Therefore, we analyzed the psychometric properties of the YPQ in terms of factor structure, internal consistency, and construct validity. To this end, CFA was conducted to test the fit of the two-factor model proposed by the Mental Health Commission of Canada (MHCC; Stuart et al., 2014). In addition, we tested a two-factor model with method effects (model 2 as respecification of model 1) that incorporated correlated error terms on the negatively phrased items. The weighted least squares mean and variance (WLSMV) was used as an estimation method to test the fit of the factor models. The following indices were examined to evaluate model fit (Schumacker and Lomax, 2010): chi-square (non-significance reflects good fit), the Tucker-Lewis index (TLI ≥ 0.9), the comparative fit index (CFI ≥ 0.9), and the root means square error of approximation (RMSEA ≤ 0.08). Internal reliability of the YPQ and its subscales was explored with Cronbach’s α coefficient. Test–retest reliability was also evaluated in the Control group by means of the intraclass correlation coefficient. Pearson correlation with RIBS was used to explore the convergent validity of the YPQ.

Regarding the effectiveness of the “What’s up!” intervention, to examine differences in primary and secondary outcome measures between the intervention and control groups, mixed-effects models (Gueorguieva and Krystal, 2004) were performed using Restricted Maximum Likelihood (REML) to estimate the parameters. One of the advantages of mixed models is that these methods account for the correlation between the repeated measurements (baseline, post-intervention, follow-up) for an individual. Since different high schools were assigned to study conditions, the statistical analyses were computed at the group level, where mixed effects models were used to account for random subject effects within each high school. Age, gender, and items concerning previous contact with people with a mental disorder, and having experienced a mental disorder were included as covariates. We tested the effect of a treatment group (intervention vs. control), time, and the interaction term (Group × Time). A separate model was estimated for each of the four outcome measures. Analyses were conducted separately for intervention completers and for the intent-to-treat (ITT) sample. We used multiple-imputation to impute the missing values.

CFA was conducted using the statistical software MPlus 7.4. For the other analyses, SPSS v.22 and STATA13.1 were used.

Results

Participants’ Characteristics

Figure 1 shows the study flowchart. A total of 446 students were recruited and non-randomly assigned to the two study conditions. Their ages ranged from 14 to 17 years old (16 in the intervention group). Baseline evaluation was carried out in 393 students (88.12% of the initial sample); post-intervention evaluation in 367 students (82.29%) and follow-up was conducted in 385 students (86.32%). As these evaluations were carried out in the ordinary classrooms, absent students did not undergo the corresponding evaluation.

FIGURE 1

FIGURE 1. Flow chart of the “What’s up!” study; distribution of sample size by high school.

We did not find significant differences in baseline characteristics between study arms (see Table 2). Distribution by gender was very similar (52% women) and the mean age was approximately 14 years old. A fifth of the students (21%) had experienced some kind of mental health problem.

TABLE 2

TABLE 2. Baseline characteristics of the sample by condition (Intervention and Control).

No statistically significant differences were found between those students who answered at follow-up and those who did not for either the entire sample or both groups considered separately (all p > 0.05).

Psychometric Analysis of the YPQ

CFA yielded significant χ² values for both models (model 1 = 727.86, p < 0.01; model 2 = 434.99, p < 0.001). The other fit indices of model 2 indicated a more adequate fit to the data (CFI = 0.936; TLI = 0.924; RMSEA = 0.058, 90 and CI 0.051–0.065) than the first model (CFI = 0.863; TLI = 0.848; RMSEA = 0.082, 90 and CI 0.075–0.088). The reliability of the original version of the YPQ-SS was 0.79 and for YPQ-SAS it was 0.85 (Stuart et al., 2014).

In Model 2, the statistically significant correlated residuals were as follows: θ_14-19= 0.29; θ_14-20= 0.18; θ_14-21= 0.23; θ_14-22= 0.21; θ_19-20= 0.53; θ_19-21= 0.33; θ_20-21= 0.57; θ_20-22= 0.33; θ_21-22= 0.36. A table with all the factor loadings can be found as Supplementary Material.

The overall internal consistency of the YPQ in our sample, based on Cronbach’s alpha among all items, was 0.84. We also calculated internal consistency at the scale level using the α “if item deleted” option. The alpha value for the total score ranged from 0.83 to 0.85. Regarding YPQ sub-scales, both present acceptable values (α = 0.73 for YPQ-SS and α = 0.8 for YPQ-SAS). Examining the α “if item deleted” option, the YPQ-SS and YPQ-SAS obtained values that ranged from 0.69 to 0.72; and 0.77 to 0.81, respectively. The test–retest reliability for the YPQ total score was 0.84, while the YPQ-SS and the YPQ-SAS both obtained 0.80. Convergent validity was also satisfactory as statistically significant correlations were found between the YPQ and its two factors, and the other measure used in this study (RIBS). The YPQ global score obtained correlations of r = 0.63 with RIBS; the YPQ-SS presented lower correlations, yet significant (r = 0.47). Correlations of r = 0.62 with RIBS were found for the YPQ-SAS.

Effectiveness of the “What’s Up!” Intervention

Table 3 presents descriptive statistics for the study measures by group throughout the study. Statistical analyses revealed that the interaction Group (Intervention vs. Control) × Time (pre, post, and follow up) was significant for all measures (all p < 0.01). Table 4 shows all the fixed and random effects of the models, which proved to be statistically significant for all measures (Chi squared: YPQ-SS = 259.42, p < 0.01; YPQ-SAS = 297.38, p < 0.01; RIBS = 217.02, p < 0.01). These interaction results imply that the groups differed in rate and manner of change over the course of the study. The YPQ sub-scale scores and RIBS scores in the intervention group improved significantly from baseline to post-intervention (p < 0.01 for YPQ-SS and p = 0.03 for YPQ-SAS; p = 0.01 for RIBS) and from baseline to follow-up (p < 0.01 for YPQ-SS and p = 0.01 for YPQ-SAS; p = 0.02 for RIBS). Interestingly, no statistically significant difference was found between the post-intervention evaluation and follow-up (all p > 0.05), which indicates that the changes observed after the intervention were maintained over time. In contrast, no differences were found from baseline to post-treatment in the control group (p = 0.06 for YPQ-SS, p = 0.16 for YPQ-SAS; p = 0.10 for RIBS) or between baseline and follow-up (p = 0.08 for YPQ-SS and p = 1.00 for YPQ-SAS; p = 1.00 for RIBS). As can be observed in Figure 2 and Table 3, with the exception of the YPQ-SS, all measures worsened at the post-treatment evaluation and returned to baseline values at follow-up.

TABLE 3

TABLE 3. Mean scores and standard deviations for the study measures (YPQ and RIBS).

TABLE 4

TABLE 4. Mixed effects models for YPQ-SS, YPQ-SAS, and RIBS (completers’ approach).

FIGURE 2

FIGURE 2. Mean scores and standard deviations for YPQ-SS (A), YPQ-SAS (B), and RIBS (C) by group throughout the study. Black and continuous line is for intervention group. Gray and dotted line is for control group. Higher scores in YPQ and RIBS represent higher stigma levels.

Despite the significant interactions Group × Time, post hoc analysis revealed that groups did not differ significantly in any of the study periods in any of the study outcomes (all p > 0.05), which indicates that the change achieved by the intervention was small.

Analysis of covariates indicated that both stereotypical attitudes and intended behavior were significantly better in women than in men (all p < 0.01), and attitudes, measured with the YPQ-SS, were less stigmatizing in those students who had a close relative or a friend with mental health problems (p = 0.03; p = 0.04, respectively). Intended behavior measured with RIBS also presented significant effects of the covariates “having a close relative with a mental disorder” (p = 0.01), “having a friend with a mental disorder” (p = 0.01), and “having experienced mental disorders” (p = 0.01). The other covariates did not present significant effects for this scale.

Following multiple imputation of missing values (data available upon request), results were very similar to those obtained in the completers’ approach. Only some aspects changed: the effect of the follow-up measure became significant for the YPQ-SS (p < 0.01) but, surprisingly, the interaction Group × Time lost its significance at follow-up for the YPQ-SAS and the RIBS (p = 0.26 and p = 0.15, respectively). Furthermore, some of the covariates which presented no significant effect changed after the multiple imputation of missing values, such as age, which became significant for all the outcome measures (all p < 0.01).

Discussion

The main objective of this study was to analyze the effectiveness of the “What’s up!” intervention on mental illness stigma in a large adolescent Catalan sample and, secondly, to examine, for the first time, the psychometric properties of the YPQ in a Catalan-speaking sample. The results can be summarized as follows: statistically significant but small improvements were found in the intervention group, and the psychometric analysis of the main outcome measure (YPQ) revealed a two-factor structure, good reliability coefficients for both dimensions (YPQ-SS and YPQ-SAS), and adequate convergent validity.

To our knowledge, this is the first study to report a full validation of the YPQ, as the original authors only assessed its internal reliability. A CFA was conducted, assuming that the two YPQ scales (i.e., YPQ-SS and YPQ-SAS) described by the authors would imply a two-factor structure. This analysis indicated that the YPQ shows a two-factor structure and similar psychometric properties to those reported by the designers of the measure (Stuart et al., 2014). A common result in CFA of psychological instruments composed of direct and reverse scored items is to obtain an inadequate fit because positively phrased items are prone to load on one factor and negatively phrased items on another (Woods, 2006). We decided to covariate the error terms of the reverse scored items to resolve this methodological problem and we obtained a better fit than without covariance. The YPQ had good internal consistency and convergent validity with the RIBS. When taking the questionnaires’ two scales separately, both correlated significantly with the RIBS. All items loaded correctly on their respective factor with the exception of item 4 (“People with a mental illness could snap out of it if they wanted to”), which presented a very low λ for both models.

Without ignoring that our sample may not reflect the whole variability of adolescence as its age variability is very low, our results show that the YPQ is a valuable instrument for use in further studies addressed to secondary students. Its value lies not only in its sound psychometric properties but also in the adequacy of the language and the aspects evaluated, which are directly related to the reality of adolescence (e.g., sitting next to a classmate with a mental disorder or being taught by a teacher with mental health problems). This favors the YPQ in comparison with other commonly used measures such as the Community Attitudes toward Mental Illness (CAMI; Taylor and Dear, 1981), whose items contain questions related to the health system or the economic costs of people with a mental disorder, areas which are not often well-known by teenagers. Moreover, the YPQ items reflect the realities of today’s teenager, such as bullying or volunteering. Many measures were designed for previous studies, but not all of them were reliable and valid (Sakellari et al., 2011). Consequently, the present study provides validation of the YPQ to help overcome the difficulty of choosing which measure to use when delivering an anti-stigma intervention to a teenage population.

Regarding the effects of the intervention, we can conclude that “What’s up!” was able to improve the attitudes and intended behavior of the students. Despite being significant, the changes achieved by the intervention were small, similar to those observed in some previous studies (Mino et al., 2001; Altindag et al., 2006; Kerby et al., 2008; Lincoln et al., 2008), although the reason why these interventions did not have a bigger impact was considered to be the fact that they were addressed to medical students, a population with such high levels of knowledge about mental disorders that they focused more on the negative aspects of these conditions, making their stigma stronger (Corrigan and Watson, 2007; Angermeyer et al., 2011). Nonetheless, this cannot be considered the reason why “What’s up!” has not presented more significant effects on reducing stigma, as our sample was constituted of high school students who did not have such a degree of knowledge. In all likelihood, the fact that the students already presented very low levels of stigma before the intervention (floor effect) was in part responsible for the small effects of the intervention. This is in line with previous findings showing the Spanish population reporting lower levels of stigma than those of other European and American countries (Aznar-Lou et al., 2016), which would imply that it could be more efficient to address interventions such as “What’s up!” to students who report high levels of stigma.

To our knowledge, this is the first time that the effectiveness of a curricular intervention addressed to fight stigma toward mental disorders has been explored. Previous approaches were commonly based on short but very immersive workshops which used materials such as presentations, videos, role-plays, talks, and contact with people with mental disorders. Some were reported to have achieved a greater impact on stigma than “What’s up!” (Petchers et al., 1988; Mound and Butterill, 1993; Esters et al., 1998; Rahman et al., 1998; Ng and Chan, 2002; Pinfold et al., 2003; Schulze et al., 2003; Watson et al., 2004; Stuart, 2006; Spagnolo et al., 2008; Essler et al., 2009), mostly on knowledge about mental health problems (Sakellari et al., 2011), an aspect that was not evaluated in our study. However, many of these studies had methodological shortcomings such as the absence of a control group (Mound and Butterill, 1993; Pinfold et al., 2003; Stuart, 2006; Essler et al., 2009) and only three conducted a follow-up, so these results should be considered with caution. Additionally, a very common limitation of previous interventions was that they seldom reported sustained improvements over time (Economou et al., 2012; Thornicroft et al., 2016), which implied that they had short-term effects that were not maintained at follow-up. In contrast, our results indicate that the improvements in stigmatizing attitudes and intended behavior achieved by “What’s up!” were maintained at 9-month follow-up. It is possible that the model of intervention used for this campaign, where the teachers include the anti-stigma material in their classes, makes it easier for the students to get involved with the information than in those interventions in which an external professional brings the material to the adolescents in a workshop format. This may produce a greater immediate reduction in stigma as it is more immersive than doing their regular classes with some didactic units on mental health, but its effects may be more superficial and easily lost with the passage of time.

These results suggest that the debate on which approach is more effective to fight stigma is open. The literature provides different examples of interventions addressing this issue (workshops, whole-school approaches, and now, curricular interventions). Considering the characteristics of the target population and the available resources to conduct the intervention, the most adequate alternative is chosen. In the case of curricular interventions such as “What’s up!,” our results suggest that they can be effective in promoting sustained effects regarding stigmatizing attitudes and intended behavior toward people with mental disorders. These improvements have been found in a sample with relatively low levels of stigma, which may imply that other populations with similar levels may benefit from the intervention as well. Since the results of “What’s up!” were maintained after a 9-month follow-up, it may be also considered as an affordable extension to other approaches that have already proved to be effective but only in the short-term.

In another vein, it is also interesting to discuss the significant influence of the covariates in the effectiveness findings. For instance, being female was a significant covariate for all the measures, which is in line with previous studies (Stickney et al., 2012). Some studies have found that adolescent girls had more knowledge and experience about mental disorders, reported less stigma toward people with them and felt more likely to use mental health services (Barrett and Raskin White, 2003). The statistical analysis did not find a significant effect of being female on the response to the intervention, but this effect has been found in previous studies (Thornicroft et al., 2007). A possible explanation for this could be that traditional ideals of masculinity (e.g., success, power, competition, emotional repression) influence teenage boys in such a way that messages designed to combat stigma with respect to mental disorders have a smaller impact than they do on teenage girls (Chandra and Minkovitz, 2006).

Two other covariates that also appeared to be significant in all the measures but the YPQ-SAS were having a close relative with a mental disorder and having a friend with a mental disorder. Again, no significant effect of these variables on the response to the intervention was found, so it cannot be concluded that these adolescents are more likely to decrease their stigma. However, it can be asserted that their levels of stigma were significantly lower than those reported by the teenagers who did not have friends or close relatives with a mental disorder. It may be the case that having a close relationship with someone with a mental disorder makes it easier for teenagers to feel empathy with them and so to decrease their levels of stigma more significantly than adolescents who have no friends or close relatives with this kind of problem, as it has been observed that empathy is a key individual factor influencing attitude change toward mental disorders (Couture and Penn, 2003).

Strengths and Limitations

To our knowledge, this is the first study to report the effectiveness of a curricular intervention to combat stigma associated with mental disorders. This is a controlled study, but not randomized, so there could be some selection bias. In this respect, it is notable that the comparative analyses in the basal phase showed no significant differences between the intervention and the control group in any of the study measures. Due to the low number of independent variables of this study, no mediation analysis was conducted.

Some limitations about the intervention itself must be considered: on the one hand, the instructions given to the teachers were very flexible, requiring the application of just three of the nine didactic units proposed by the project, although remarking that the first and the last (which contained the social contact component of the campaign) were considered “essential.” It can be concluded that a little more systematic orientation in applying the intervention is needed to evaluate intervention effectiveness, although it should be noted that all high schools implemented at least five of the nine didactic units and an important strength of the intervention was that this degree of flexibility made “What’s up!” a truly implementable intervention in the real context of high schools.

We must take into account that the timing was not the same for all the high schools at the post-intervention measure. Although all started within 2 weeks of each other, one class took the post-intervention measure only 25 days after baseline assessment, while the rest of the classes were evaluated between 77 and 105 days after baseline. These differences may have had an impact on the results as it seems reasonable to suppose that the more time spent on didactic units in class, the more impact they would have on the students. This could be easily corrected by adjusting the evaluation periods and conducting assessments at roughly the same time in all schools. Additionally, during application of the intervention there was no register of external events that could interfere with it (e.g., to include an item in the socio-demographic scale registering contact with fictional characters with mental health problems either in movies, TV shows, or novels).

Regarding the external validity of the intervention, it should be borne in mind that, although the measures used for this study presented strong psychometric properties, their items might be susceptible to social desirability and more objective measures should be developed to ensure that the assessment of stigma is as close to actual behavior as possible. It would also have been interesting to include a qualitative measure in the evaluation of the impact of the intervention to determine which aspects had the biggest effect on the participants and why. This could have been especially useful considering that “What’s up!” consisted of nine distinct units implemented by different teachers, which, as mentioned previously, introduced a high degree of variation to the intervention, and not all the units could impact equally for every student. Future studies with a larger sample size could use a “dismantling design” to examine which of the multiple didactic units included in the “What’s up!” intervention are the active ingredients of change. One typical method for conducting this type of study is to compare single intervention components with the global intervention.

As noted above, an important strength of this study is the use of reliable and valid outcome measures. The YPQ emerged as a useful instrument that measures both attitudes and intended behavior and whose psychometric attributes have been found to be positive. However, it is true that its convergent validity was only studied with RIBS, which is an intended behavior measure, and it would also be interesting to study it with a measure of stigmatizing attitudes such as the CAMI. It could also be argued that this measure is not very sensitive to change, but as similar results were found for RIBS, we must conclude that the intervention did not have the impact required to produce more significant changes, maybe due to some of its previously explained limitations. Moreover, we should point out that our results may not be totally generalizable to the whole adolescent population, as our sample presented a very low degree of age variability.

Conclusion

“What’s up!” is a promising, original intervention model in which teachers are the responsible for introducing campaign materials into regular didactic units, without modifying the school curriculum and offering the students the opportunity to learn about mental health through education and social contact. Although significant improvements were observed, these results were small. To overcome the shortcomings of this intervention, further anti-stigma interventions could focus on reducing time differences between classes and registering external variables that could affect students’ stigma levels. Additionally, it may be more appropriate to address the intervention especially to those students who report higher levels of stigma at baseline to counter the “floor effect.”

On the other hand, the present work has satisfactorily validated the YPQ, an instrument that had only been used in one previous intervention and whose psychometric properties had not yet been reported. This questionnaire evaluates stereotypical attitudes and intended behavior and presents good internal consistency, test–retest reliability and convergent validity and can thus be considered as a valuable measurement instrument for use in further studies of stigma in adolescent populations.

Author Contributions

AP-A and LA-R made substantial contribution to the analysis and to the interpretation of the data, drafted the manuscript, provided final approval of the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. JL, IA-L, MR-V, and AF-S made substantial contributions to the conception and the design of the study, drafted the manuscript, provided final approval of the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. AS-B helped out in the interpretation of data for the work, revised the manuscript critically for important intellectual content, provided final approval of the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. MJ and AT revised the manuscript critically for important intellectual content, provided final approval of the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. AB revised the manuscript critically for important intellectual content, provided final approval of the version to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding

We are grateful to the Instituto de Salud Carlos III (ISCIII) through the Network for Prevention and Health Promotion in Primary Care (RD16/0007/0012). The third listed author (AF-S) has a “Sara Borrell” research contract from the ISCIII (CD16/00147). The last listed author (JL) has a “Miguel Servet” research contract from the ISCIII (CP14/00087). Co-financed with European Union ERDF funds. We disclose receipt of the following financial support for the research, authorship, and/or publication of this article: the OBERTAMENT campaign was supported by the “Obra Social La Caixa.”

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgment

Thanks to Helena Quesada, Gemma Cantoni, and Aleix Caussa for their invaluable collaboration in the development of “What’s up!,” and to Stephen Kelly for English editing.

Supplementary Material

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg.2017.01608/full#supplementary-material

References

Altindag, A., Yanik, M., Ucok, A., Alpetkin, K., and Ozkan, M. (2006). Effects of an antistigma program on medical students’ attitudes towards people with schizophrenia. Psychiatry Clin. Neurosci. 60, 283–288. doi: 10.1111/j.1440-1819.2006.01503.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Ando, S., Yamaguchi, S., Aoki, Y., and Thornicroft, G. (2013). Review of mental-health-related stigma in Japan. Psychiatry Clin. Neurosci. 67, 471–482. doi: 10.1111/pcn.12086

PubMed Abstract | CrossRef Full Text | Google Scholar

Angermeyer, M. C., Holzinger, A., Carta, M. G., and Schomerus, G. (2011). Biogenetic explanations and public acceptance of mental illness: systematic review of population studies. Br. J. Psychiatry 199, 367–372. doi: 10.1192/bjp.bp.110.085563

PubMed Abstract | CrossRef Full Text | Google Scholar

Aznar-Lou, I., Serrano-Blanco, A., Fernández, A., Luciano, J. V., and Rubio-Valera, M. (2016). Attitudes and intended behaviour to mental disorders and associated factors in catalan population, Spain: cross-sectional population-based survey. BMC Public Health 16:127. doi: 10.1186/s12889-016-2815-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Barrett, A. E., and Raskin White, H. (2003). Trajectories of gender role orientations in adolescence and early adulthood: a prospective study of the mental health effects of masculinity and femininity. J. Health Soc. Behav. 43, 451–468. doi: 10.2307/3090237

PubMed Abstract | CrossRef Full Text | Google Scholar

Batterham, P. J., Griffiths, K. M., Barney, L. J., and Parsons, A. (2013). Predictors of generalized anxiety disorder stigma. Psychiatry Res. 206, 282–286. doi: 10.1016/j.psychres.2012.11.018

PubMed Abstract | CrossRef Full Text | Google Scholar

Bulanda, J. J., Bruhn, C., Byro-Johnson, T., and Zentmyer, M. (2014). Addressing mental health stigma among young adolescents: evaluation of a youth-led approach. Health Soc. Work 39, 73–80. doi: 10.1093/hsw/hlu008

PubMed Abstract | CrossRef Full Text | Google Scholar

Chandra, A., and Minkovitz, C. (2006). Stigma starts early: gender differences in teen willingness to use mental health services. J. Adolesc. Health 38, 754.e1–754.e8. doi: 10.1016/j.jadohealth.2005.08.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Coppens, E., Van Audenhove, C., Scheerder, G., Arensman, E., Coffey, C., Costa, S., et al. (2013). Public attitudes toward depression and help-seeking in four European countries baseline survey prior to the OSPI-Europe intervention. J. Affect. Disord. 150, 320–329. doi: 10.1016/j.jad.2013.04.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Corrigan, P. W., Morris, S. B., Michaels, P. J., Rafacz, J. D., and Rüsch, N. (2012). Challenging the public stigma of mental illness: a meta-analysis of outcome studies. Psychiatr. Ser. 63, 963–973. doi: 10.1176/appi.ps.201100529

PubMed Abstract | CrossRef Full Text | Google Scholar

Corrigan, P. W., and Penn, D. L. (1999). Lessons from social psychology on discrediting psychiatric stigma. Am. Psychol. 54, 765–776. doi: 10.1037/0003-066X.54.9.765

PubMed Abstract | CrossRef Full Text | Google Scholar

Corrigan, P. W., and Shapiro, J. R. (2010). Measuring the impact of programs that challenge the public stigma of mental illness. Clin. Psychol. Rev. 30, 907–922. doi: 10.1016/j.cpr.2010.06.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Corrigan, P. W., and Watson, A. C. (2007). How children stigmatize people with mental illness. Int. J. Soc. Psychiatry 53, 526–546. doi: 10.1177/0020764007078359

PubMed Abstract | CrossRef Full Text | Google Scholar

Couture, S. M., and Penn, D. L. (2003). Interpersonal contact and the stigma of mental illness: a review of the literature. J. Ment. Health 12, 291–305. doi: 10.1080/09638231000118276

PubMed Abstract | CrossRef Full Text | Google Scholar

Crespo, M., Pérez-Santos, E., Muñoz, M., and Guillén, A. I. (2008). Descriptive study of stigma associated with severe and persistent mental illness among the general population of Madrid (Spain). Community Ment. Health J. 44, 393–403. doi: 10.1007/s10597-008-9142-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Crisp, A. H., Gelder, M. G., Rix, S., Meltzer, H. I., and Rowlands, O. J. (2000). Stigmatisation of people with mental illnesses. Br. J. Psychiatry 177, 4–7. doi: 10.1192/bjp.177.1.4

CrossRef Full Text | Google Scholar

Des Jarlais, D. C., Lyles, C., Crepaz, N., and Trend Group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am. J. Public Health 94, 361–366. doi: 10.2105/AJPH.94.3.361

PubMed Abstract | CrossRef Full Text | Google Scholar

Economou, M., Louki, E., Peppou, L. E., Gramandani, C., Yotis, L., and Stefanis, C. N. (2012). Fighting psychiatric stigma in the classroom: the impact of an educational intervention on secondary school students’ attitudes to schizophrenia. Int. J. Soc. Psychiatry 58, 544–551. doi: 10.1177/0020764011413678

PubMed Abstract | CrossRef Full Text | Google Scholar

Essler, V., Arthur, A., and Stickley, T. (2009). Using a school-based intervention to challenge stigmatizing attitudes and promote mental health in teenagers. J. Ment. Health 15, 243–250. doi: 10.1080/09638230600608669

CrossRef Full Text | Google Scholar

Esters, I. G., Cooker, P. G., and Ittenbach, R. F. (1998). Effects of a unit of instruction in mental health on rural adolescents’ conceptions of mental illness and attitudes about seeking help. Adolescence 33, 469–476.

Google Scholar

Evans-Lacko, S., Rose, D., Little, K., Flach, C., Rhydderch, D., Henderson, C., et al. (2011). Development and psychometric properties of the reported and intended behaviour scale (RIBS): a stigma-related behaviour measure. Epidemiol. Psychiatr. Sci. 20, 263–271.

PubMed Abstract | Google Scholar

Flavell, J. H., Miller, P. H., and Miller, S. A. (2002). Cognitive Development, 4th Edn. Upper Saddle River, NJ: Prentice Hall.

Google Scholar

Gueorguieva, R., and Krystal, J. H. (2004). Move Over ANOVA progress in analyzing repeated-measures data and its reflection in papers published in the archives of general psychiatry. Arch. Gen. Psychiatry 61, 310–317. doi: 10.1001/archpsyc.61.3.310

PubMed Abstract | CrossRef Full Text | Google Scholar

Hansson, L., and Markström, U. (2014). The effectiveness of an anti-stigma intervention in a basic police officer training programme: a controlled study. BMC Psychiatry 14:55. doi: 10.1186/1471-244X-14-55

PubMed Abstract | CrossRef Full Text | Google Scholar

Haro, J., Palacín, C., Vilagut, G., Romera, B., Codony, M., Autonell, J., et al. (2003). La epidemiología de los trastornos mentales en España: métodos y participación en el proyecto ESEMeD-España. Actas Esp. Psiquiatr. 31, 182–191.

Google Scholar

Hing, N., and Russell, A. M. T. (2017). How anticipated and experienced stigma can contribute to self-stigma: the case of problem gambling. Front. Psychol. 8:235. doi: 10.3389/fpsyg.2017.00235

PubMed Abstract | CrossRef Full Text | Google Scholar

Hugo, C. J., Boshoff, D. E. L., Traut, A., Zungu-Dirwayi, N., and Stein, D. J. (2003). Community attitudes toward and knowledge of mental illness in South Africa. Soc. Psychiatry Psychiatr. Epidemiol. 38, 715–719. doi: 10.1007/s00127-003-0695-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Ke, S., Lai, J., Sun, T., Yang, M. M. H., Wang, J. C. C., and Austin, J. (2015). Healthy young minds: the effects of a 1-hour classroom workshop on mental illness stigma in high school students. Community Ment. Health J. 51, 329–337. doi: 10.1007/s10597-014-9763-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Kerby, J., Calton, T., Dimambro, B., and Glazebrook, C. (2008). Anti-stigma films and medical student’s attitudes towards mental illness and psychiatry: randomised controlled trial. Psychiatr. Bull. 32, 345–349. doi: 10.1192/pb.bp.107.017152

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, J., Li, J., Thornicroft, G., and Huang, Y. (2014). Levels of stigma among community mental health staff in Guangzhou, China. BMC Psychiatry 14:231. doi: 10.1186/s12888-014-0231-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Lincoln, T. M., Arens, E., Berger, C., and Rief, W. (2008). Can antistigma campaigns be improved? A test of the impact of biogenetic vs psychosocial causal explanations on implicit and explicit attitudes to schizophrenia. Schizophr. Bull. 34, 984–994. doi: 10.1093/schbul/sbm131

PubMed Abstract | CrossRef Full Text | Google Scholar

Livingston, J. D., and Boyd, J. E. (2010). Correlates and consequences of internalized stigma for people living with mental illness: a systematic review and meta-analysis. Soc. Sci. Med. 71, 2150–2161. doi: 10.1016/j.socscimed.2010.09.030

PubMed Abstract | CrossRef Full Text | Google Scholar

Loch, A. A., Wang, Y.-P., Guarniero, F. B., Lawson, F. L., Hengartner, M. P., Rössler, W., et al. (2014). Patterns of stigma toward schizophrenia among the general population: a latent profile analysis. Int. J. Soc. Psychiatry 60, 595–605. doi: 10.1177/0020764013507248

PubMed Abstract | CrossRef Full Text | Google Scholar

Mino, Y., Yasuda, N., Tsuda, T., and Shimodera, S. (2001). Effects of a one-hour educational program on medical students’ attitudes to mental illness. Psychiatry Clin. Neurosci. 55, 501–507. doi: 10.1046/j.1440-1819.2001.00896.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Mound, B., and Butterill, D. (1993). Beyond the Cuckoo’s nest: a high school education program. Psychosoc. Rehabil. J. 16, 146–150. doi: 10.1037/h0095666

CrossRef Full Text | Google Scholar

Ng, P., and Chan, K.-F. (2002). Attitudes towards people with mental illness. Effects of a training program for secondary school students. Int. J. Adolesc. Med. Health 14, 215–224. doi: 10.1515/IJAMH.2002.14.3.215

CrossRef Full Text | Google Scholar

Nunnally, J. C. (1978). Psychometric Theory, 2nd Edn. New York: McGraw-Hill.

Google Scholar

Ochoa, S., Martínez-Zambrano, F., Vila-Badia, R., Arenas, O., Casas-Anguera, E., García-Molares, E., et al. (2015). Validación al castellano de la escala de estigma social: community attitudes towards mental illness en población adolescente. Rev. Psiquiatr. Salud Ment. 73, 150–157. doi: 10.1016/j.rpsm.2015.02.002

PubMed Abstract | CrossRef Full Text

Petchers, M. K., Biegel, D. E., and Drescher, R. (1988). A video-based program to educate high school students about serious mental illness. Psychiatr. Serv. 39, 1102–1103. doi: 10.1176/ps.39.10.1102

PubMed Abstract | CrossRef Full Text | Google Scholar

Pinfold, V., Toulmin, H., Thornicroft, G., Huxley, P., Farmer, P., and Graham, T. (2003). Reducing psychiatric stigma and discrimination: evaluation of educational interventions in UK secondary schools. Br. J. Psychiatry? 182, 342–346. doi: 10.1192/bjp.182.4.342

PubMed Abstract | CrossRef Full Text | Google Scholar

Polanczyk, G. V., Salum, G. A., Sugaya, L. S., Caye, A., and Rohde, L. A. (2015). Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J. Child Psychol. Psychiatry 56, 345–365. doi: 10.1111/jcpp.12381

PubMed Abstract | CrossRef Full Text | Google Scholar

Promoting Children and Young People’s Emotional Health and Wellbeing (2015). Available at: www.gov.uk/phe

Rahman, A., Mubbashar, M. H., Gater, R., and Goldberg, D. (1998). Randomised trial of impact of school mental-health programme in rural Rawalpindi, Pakistan. Lancet 352, 1022–1025. doi: 10.1016/S0140-6736(98)02381-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Rüsch, N., Angermeyer, M. C., and Corrigan, P. W. (2005). Mental illness stigma: concepts, consequences, and initiatives to reduce stigma. Eur. Psychiatry 20, 529–539. doi: 10.1016/j.eurpsy.2005.04.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Rüsch, N., Zlati, A., Black, G., and Thornicroft, G. (2014). Does the stigma of mental illness contribute to suicidality? Br. J. Psychiatry 205, 257–259. doi: 10.1192/bjp.bp.114.145755

PubMed Abstract | CrossRef Full Text | Google Scholar

Sakellari, E., Leino-Kipli, H., and Kalokerinou-Anagnostopoulo, A. (2011). Educational interventions in secondary education aiming to affect pupils’ attitudes towards mental illness: a review of the literature. J. Psychiatr. Ment. Health Nurs. 18, 166–176. doi: 10.1111/j.1365-2850.2010.01644.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Schulze, B., Richter-Werling, M., Matschinger, H., and Angermeyer, M. C. (2003). Crazy? So what! Effects of a school project on students’ attitudes towards people with schizophrenia. Acta Psychiatr. Scand. 107, 142–150. doi: 10.1034/j.1600-0447.2003.02444.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Schumacker, R. E., and Lomax, R. G. (2010). A Beginner’s Guide to Structural Equation Modeling. New York, NY: Routledge.

Google Scholar

Sharac, J., McCrone, P., Clement, S., and Thornicroft, G. (2010). The economic impact of mental health stigma and discrimination: a systematic review. Epidemiol. Psichiatr. Soc. 19, 223–232. doi: 10.1017/S1121189X00001159

CrossRef Full Text | Google Scholar

Spagnolo, A. B., Murphy, A. A., and Librera, L. A. (2008). Reducing stigma by meeting and learning from people with mental illness. Psychiatr. Rehabil. J. 31, 186–193. doi: 10.2975/31.3.2008.186.193

PubMed Abstract | CrossRef Full Text | Google Scholar

Stickney, S., Yanosky, D., Black, D. R., and Stickney, N. L. (2012). Socio-demographic variables and perceptual moderators related to mental health stigma. J. Ment. Health 21, 244–256. doi: 10.3109/09638237.2012.670878

PubMed Abstract | CrossRef Full Text | Google Scholar

Stuart, H. (2006). Reaching out to high school youth: the effectiveness of a video-based antistigma program. Can. J. Psychiatry 51, 647–653. doi: 10.1177/070674370605101004

PubMed Abstract | CrossRef Full Text | Google Scholar

Stuart, H., Chen, S. P., Christie, R., Dobson, K., Kirsh, B., Knaak, S., et al. (2014). Opening minds in Canada: targeting change. Can. J. Psychiatry 59(10 Suppl. 1), S13–S18. doi: 10.1177/070674371405901S05

PubMed Abstract | CrossRef Full Text | Google Scholar

Taylor, S. M., and Dear, M. J. (1981). Scaling community attitudes toward the mentally ill. Schizophr. Bull. 7, 225–240.

Google Scholar

Thornicroft, G., Mehta, N., Clement, S., Evans-Lacko, S., Doherty, M., Rose, D., et al. (2016). Evidence for effective interventions to reduce mental-health-related stigma and discrimination. Lancet 387, 1123–1132. doi: 10.1016/S0140-6736(15)00298-6

CrossRef Full Text | Google Scholar

Thornicroft, G., Rose, D., Kassam, A., and Sartorius, N. (2007). Stigma: ignorance, prejudice or discrimination? Br. J. Psychiatry 190, 192–193.

Google Scholar

Warner, R. (2005). Local projects of the world psychiatric association programme to reduce stigma and discrimination. Psychiatr. Serv. 56, 570–575. doi: 10.1176/appi.ps.56.5.570

PubMed Abstract | CrossRef Full Text | Google Scholar

Watson, A. C., Otey, E., Westbrook, A. L., Gardner, A. L., Lamb, T., Corrigan, P. W., et al. (2004). Changing middle schoolers’ attitudes about mental illness through education. Schizophr. Bull. 30, 563–572. doi: 10.1093/oxfordjournals.schbul.a007100

CrossRef Full Text | Google Scholar

Wittchen, H. U., Jacobi, F., Rehm, J., Gustavsson, A., Svensson, M., Jönsson, B., et al. (2011). The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur. Neuropsychopharmacol. 21, 655–679. doi: 10.1016/j.euroneuro.2011.07.018

PubMed Abstract | CrossRef Full Text | Google Scholar

Woods, C. M. (2006). Careless responding to reverse-worded items: implications for confirmatory factor analysis. J. Psychopathol. Behav. Assess. 28, 186–191. doi: 10.1007/s10862-005-9004-7

CrossRef Full Text | Google Scholar

Yamaguchi, S., Mino, Y., and Uddin, S. (2011). Strategies and future attempts to reduce stigmatization and increase awareness of mental health problems among young people: a narrative review of educational interventions. Psychiatry Clin. Neurosci. 65, 405–415. doi: 10.1111/j.1440-1819.2011.02239.x

PubMed Abstract | CrossRef Full Text | Google Scholar