Controlled Study of the Impact of a Virtual Program to Reduce Stigma Among University Students Toward People With Mental Disorders

Stigma toward mental disorders is one of today's most pressing global issues. The Covid-19 pandemic has exacerbated the barriers to social inclusion faced by individuals with mental disorders. Concurrently, stigma reduction interventions, especially those aimed at university students, have been more difficult to implement given social distancing and campus closures. As a result, alternative delivery for programs contributing to stigma reduction is required, such as online implementation. This paper reports the results of a controlled study focused on an online multi-component program on reducing stigma toward mental illness that included project-based learning, clinical simulations with standardized patients and E-Contact with real patients. A total of 40 undergraduate students from the Universidad del Desarrollo in Santiago, Chile, participated in the study. They were randomly divided between an intervention and control group. The intervention group participated in the online multi-component program, while the control group participated in an online educational program on cardiovascular health. We assessed the impact of the program by using the validated Spanish-language versions of the Attribution Questionnaire AQ-27 and the Questionnaire on Student Attitudes toward Schizophrenia with both groups, before and after the intervention. In addition, an ad hoc Likert scale ranging from 0 to 5 was used with the intervention group in order to assess the learning strategies implemented. Following the intervention, the participants belonging to the intervention group displayed significantly lower levels of stereotypes, perception of dangerousness, and global score toward people with schizophrenia (p < 0.001). In addition, participants presented lower levels of dangerousness-fear, avoidance, coercion, lack of solidarity, and global score (p < 0.001). The control group displayed no statistically significant differences in the level of stigma before and after the evaluation, for all of the items assessed. Finally, the overall assessment of each of the components of the program was highly positive. In conclusion, the study shows that online programs can contribute to reducing stigma toward mental disorders. The program assessed in this study had a positive impact on all the dimensions of stigma and all of the components of the program itself were positively evaluated by the participants.

Introduction

Stigma toward mental disorders is one of today's most pressing global issues. (1), with important implications such as high levels of social exclusion (2), poor quality of life (3), and low social support (4), for those affected by them. Furthermore, there are barriers to access and continuity of education and employment (5, 6), and higher risks of dying by suicide (6) among this specific population.

Mental health professionals and medical students have been found to stigmatize psychiatric patients (7, 8), and 40% of people with a mental disorder report unfair treatment by health professionals such as medical doctors, psychiatrists, psychologists, and nurses (9, 10). Experiencing stigma in medical settings is a stressor that contributes, among others, to reducing patients' quality of life and to exacerbating inequities in health outcomes and access to healthcare (11–13). Furthermore, stigma is considered a major obstacle against processes of mental health recovery (14–16) and is a main determinant of the quality of care delivered by health professionals and medical students (17, 18).

At the international level, several strategies have been implemented within the general population, as well as medical students, to reduce stigma toward mental disorders (19), and the most successful ones have been those involving direct contact with patients (20, 21) as well as programs with strong educational components (16, 22–24). It has been shown that direct, person-to-person contact and activities where the participants and patients share thoughts and experiences around mental health are key to the success of these programs (25–27).

While these strategies have been implemented in different countries (28–30) and have led to decreasing stigma (31, 32) and promoting positive perceptions of psychiatric patients among students and professionals (33, 34), their implementation in virtual learning spaces is recent, particularly for university students (35). Several authors highlight the advantages of using multimedia resources, for instance audiovisual educational resources with standardized patients (36, 37), E-contact with psychiatric patients (38), and even the implementation of educational videogames (39, 40).

In the context of the SARS-CoV−2 pandemic, lockdowns and physical distancing recommendations, stigmatization and instances of discrimination against individuals with mental disorders have increased and have negatively impacted their well-being and quality of life (41). Additionally, the implementation of measures aimed at reducing the transmission of the virus have hampered the implementation of on-site interventions focused on the reduction of stigma toward psychiatric patients (42), and in that context, developing and implementing innovative and online alternatives is paramount.

Among such alternatives, for university students, active and interdisciplinary project-based learning, where students lead and develop a final project, has been proven useful (43, 44), as it promotes critical thinking, practical application of knowledge (45), and problem-solving skills (46). Its implementation in the training of health professionals on mental disorders is recent, however it has proven to effectively foster professional skills (47) and a comprehensive and inclusive understanding of the different social processes involved in mental health outcomes and care (48–50). As it allows students to reflect on their knowledge to implement their skills in a creative way (51), it is considered an innovative educational tool for the reduction of stigma toward mental health disorders and psychiatric patients.

In addition to project-based learning, the use of standardized patients has shown different benefits for skill training and e-learning in the field of education in psychiatry and mental health (52). Including it as part of educational intervention strategies has had a positive impact (53), particularly on skill and knowledge development, and for the promotion of a holistic understanding of mental disorders (54). However, working online rather than on-site with standardized patients in order to reduce stigma toward mental disorders is a recent, underexplored development in the field, which requires complementary educational interventions (37), along with the creation of original scripts focused on the story around experiencing the illness and the recovery process (55, 56).

Finally, E-contact, defined as “computer-mediated real-time interactions where members of different groups interact online” (57), has been used as a strategy for raising awareness and reducing prejudice among ideologically different groups (58, 59). However, its implementation in the field of stigma-reduction is new and innovative, and the existing evidence shows that it can reduce stigma toward the transgender population (60). With regards to the use of E-contact to reduce stigma toward mental disorders, to date, only one experimental study has been conducted, and has demonstrated that E-contact reduces anxiety, rage and stereotypes toward individuals with schizophrenia (38). In that sense, this confirms the relevance of including E-contact in interventions aimed at reducing stigma toward people with mental disorders, for instance through the use of synchronous videoconferencing (61).

The objective of this study is to demonstrate the effectiveness of a multi-component online intervention incorporating E-contact with mental healthcare patients, standardized patients, and a project-based learning program.

Materials and Methods

Participants

The participants were recruited within the Universidad del Desarrollo in Santiago, Chile, during “Innovation Week,” an event organized by the university, where students belonging to the first 3 years of undergraduate studies in different disciplines work on different innovative responses to local social issues.

The sample consisted of 40 university students in their first, second or third year, 32 of which studied health science degrees and the remaining 8 studied social science degrees. Eighty percent of the participants were women and 20% were men, all aged between 18 and 23 years old (X = 20.6; SD = 1.3). None of the participants had been trained in psychiatry.

The intervention group and control group were each made of 20 randomly allocated participants, with equal proportion of men and women, area of study, and no statistically significant difference with regards to age (p > 0.05).

Instruments

Questionnaire on Student Attitudes Toward Schizophrenia (QSAS)

This instrument (62) aimed at evaluating the items of stereotyping toward schizophrenia and social distancing was developed for, and implemented with, high school students in Germany, during the World Program Against Stigma and Discrimination of the World Association of Psychiatry. The current study used the validated Spanish-language version developed by Navarro et al. (63), which possesses the appropriate psychometric properties, with a Cronbach's alpha of 0.95 for both evaluated factors.

Attribution Questionnaire (AQ−27)

This instrument (64) is used to quantify stigma toward people with mental disorders among the general population. It presents the participants with the case of a person diagnosed with schizophrenia and evaluates stereotyping and prejudice through a 9-point Likert scale. The current study used the Spanish-language, shorter version with 14 items, validated by Saavedra et al. (65), which possesses the appropriate psychometric properties for its four factors. (dangerousness-fear = 0.88, lack of solidarity = 0.837, coercion = 0.864, and avoidance = 0.758).

Learning Strategies Assessment Scale

This instrument was designed specifically for the current study. Participants assessed the integration of the different intervention strategies on a scale of 0 to 5, focusing on the perceived impact on empathy, understanding of recovery and social inclusion of people affected by severe mental disorders. In addition, the degree of recommendation of the program was assessed. The scale is available as Supplementary Material.

Procedure

Prior to their participation in either group, the participants gave their written informed consent in accordance with the Declaration of Helsinki (66) and the Singapore statement on research integrity (67). The study was approved by the Ethics Committee of Universidad del Desarrollo, Santiago, Chile (protocol number: 2020–142). Once the participants were randomly allocated to the intervention and control groups, they all completed the AQ−27 and QSAS questionnaires described previously. These questionnaires were sent to the participants 15 min before the beginning of the study and were completed online. The experiment began once the participants had completed both questionnaires.

With respect to the intervention, the intervention group participated in the multi-component online program, which lasted 14 h equally distributed across two days. The program consisted of the three consecutive interventions described below:

a. Two sessions including a simulation with standardized patients with emotional, substance use and anxiety disorders and an education workshop. These common mental disorders are psychiatric comorbidities and are highly prevalent among patients with schizophrenia (68, 69). An online presentation was carried out by two trained actors focused on the main characteristics of these disorders. Previous to the presentation, the research team had designed original scripts based on the recovery process of the patient and their family environment. During every simulation, dynamic interactions took place between the trained actors and the students, who asked them, for instance, about their life-experience, how they were feeling, etc. Additionally, an educational workshop was carried out for each simulation, in order to discuss the essential aspects of the mental disorders presented during the simulation. Each session lasted 2.5 h.

b. E-contact activity with an adult diagnosed with schizophrenia, who discussed and analyzed their recovery process with the students, including the role of the healthcare system, the community and their family. This activity lasted 3.5 h.

c. Project-Based Learning. During the two days of the program, activities aimed at developing an intervention around the reduction of stigma and the promotion of social inclusion for people with mental disorders, designed and led by the students were implemented. At the end of the second day, the students had the opportunity to present, in a webinar, the interventions they had developed throughout the course of the program, to the other participants and to the members of the evaluation jury, which included the patient with whom the E-contact activity was carried out. This allowed the participants to receive feedback from an individual who had experienced a severe mental disorder. The presentations were carried out in groups, and the interventions introduced were related to the promotion of social inclusion and the rights of people affected by mental disorders in the academic and local community, with emphasis on raising awareness around stigma and education on mental health. This activity had a duration of 5.5 h, distributed between the development of the initiative, tutoring from the teaching team and the final presentations. An example of one of the projects carried out by the students is available as Supplementary Material.

The control group participated in an online educational program on cardiovascular health of the same duration.

Finally, at the end of both days, each group answered the AQ-27 and QSAS questionnaires. Additionally, the intervention group completed the Learning Strategies Assessment Scale.

Figure 1 summarizes the whole procedure of the study.

FIGURE 1

Figure 1. Participant assignment procedure to the multicomponent intervention [MCI].

Statistical Analysis

Student's t-test for independent samples was used to assess whether there were statistically significant differences between the pre- and post-intervention measures in both groups. Additionally, Cohen's d was used to determine the effect size of the intervention.

The pre- and post-measures of each group were subsequently analyzed with the use of Student's t for related samples, which was complemented with the measure of the effect size of the intervention through Cohen's d. Finally, descriptive statistics were used for the analysis of the global assessment of each one of the intervention strategies. We used SPSS 22.0 software to carry out the statistical analysis.

Results

As Table 1 shows, there is no statistical difference between the intervention and control groups prior to the intervention for any of the items of either instruments. However, statistically significant differences are observed for the group means in the post-intervention measures for both groups for all of the analyzed variables. Using Cohen's d effect size allowed us to determine a great effect size for all of the dimensions evaluated for both instruments, except for the stereotypes indicator, which displays a more conservative effect size.

TABLE 1

Table 1. Student's t-test for independent samples of pre-test and post-test differences between the intervention group and control group.

Table 2 shows the average means and standard deviations for each variable for both instruments corresponding to the intervention and control groups for each phase of the study. The analysis of the post-intervention scores of the control group shows that there is no statistically significant difference for any of the variables studied in either instruments. However, statistically significant differences are observed in the same analysis of the scores for the intervention group, for all of the scores of the questionnaires in relation to stigma in every proportion. Regarding the effect size, the program had a positive impact on stigma reduction as reflected on each one of the analyzed items and the global score of both questionnaires.

TABLE 2

Table 2. Means and standard deviations of pre-test, post-test, and Student's t-test for related samples of post-test–pre-test differences in the study variables for the intervention group and control group.

The assessment given by the participants on the intervention was measured with a Likert scale ranging from 0 to 5 points. As Table 3 shows, 95% (n = 19) of the students assessed at the top level the usefulness of the program in understanding the recovery process and promoting social inclusion of patients with severe mental disorder, while 100% (n = 20) of them rated at the top level the promotion of empathy. Additionally, 95% (n = 19) stated they would recommend this program to a peer. There was no statistically significant differences between the three interventions and their integration. Finally, positive and uniform comments were presented, also in conjunction with the following representative verbatim quotes: “The activities we carried out, helped me to have a better view of mental illness, and the use of simulated patients was useful in the experience and learning”; “I really liked the simulations and the E-Contact, it makes it much more didactic to learn and understand about mental illness” and “was a wonderful experience, it would be a huge challenge to do it on-site.”

TABLE 3

Table 3. Intervention group participants who rate the integration of learning strategies at the top level (n = 20).

Discussion

Stigma toward mental disorders is a pressing issue, considering, on the one hand, its impact on the wellbeing and quality of life of individuals with mental disorders (14–19), and on the other hand, that it is pervasive among the general population, health professionals and university students (7–10, 70–72).

At present, and to the best of our knowledge, there is no multi-component program focused on stigma toward mental disorders entirely carried out online. For this reason, as well as the lockdowns and social distancing measures implemented as a result of the SARS-CoV−2 pandemic, this study focuses on the implementation of an online program.

The results show that the intervention had a positive impact on the reduction of stigma among the intervention group. These results may be explained by the intensity of the program, the integration of multiple components, the focus given to the recovery process and the implementation of direct E-contact between mental health patients and students, all of which have been demonstrated to be key points for the implementation of a successful stigma-reduction program (54–58, 73–77). Additionally, further studies comparing the effectiveness of implementing the multicomponent program on-site rather than online, are necessary. Other studies have shown that comparable online interventions have an impact similar to those implemented on-site (27, 78).

Considering that the intervention was intensive, taking into account the number of activities carried out in a short amount of time, further qualitative studies will explore the experience of the participants and the lessons learned from the implementation of the key components, in order to adapt and adjust the program and make it easier to replicate.

It is important to emphasize that the assessment that the students made of the program was positive with respect to online implementation for each of its components, and the components focused on the promotion of empathy, social inclusion and understanding of the recovery process of severe mental disorders scored the highest, especially when integrated together. This suggests that the participants enjoyed the experience, would recommend it to peers, bringing evidence on the advantages of interventions that take into account the motivation of university students toward learning and goal-achievement, which is in turn one of the main challenges of digital education (79, 80).

In that sense, the educational intervention methods used in the current study are supported by robust theoretical and empirical evidence, and are especially adequate for online implementation, which is key in the context of the SARS-CoV-2 pandemic and the restriction of on-site interventions (36–40, 53–56). This also represents an opportunity for future integration of similar programs in university curricula, as they offer the possibility to reach remote areas, to establish contact between people with different experiences around mental disorders and between patient communities through E-contact and to generate student-led projects. Furthermore, the online modality of these programs promotes collaborative work between individuals and institutions across different countries, contributing, on the one hand, to foster cultural diversity, and on the other hand, to develop international education networks (81, 82).

In conclusion, the design and implementation of this type of interventions and educational spaces focused on diminishing stigma toward psychiatric patients contribute to normalizing these experiences among the population, by openly sharing challenges around mental health and promoting effective help-seeking and the delivery of adequate treatment to those who need it (83, 84). The study shows that online programs can promote new types of interventions aimed at reducing stigma not only in a context of mandatory social-distancing and lockdowns due to the SARS-CoV-2 pandemic, but also complementing existing on-site programs.

However, the study presents several limitations. First, the sample is relatively small, comes from one specific institution and is not representative of the general population, which may undermine the generalization of the results and limit the interpretation of the effect size. Second, the phases of the intervention were not experimentally evaluated in an independent way and the measure of stigma was only evaluated for schizophrenia, making it necessary to include other prevalent mental disorders in future research. Third, specific characteristics of the participants and possible confounding factors, which may influence the results, such as their socio-cultural background and their level of knowledge, desire to work in psychiatry and closeness to mental disorders were not evaluated. Finally, the long duration of the program may limit its replication, presenting several logistical challenges in its implementation through virtual platforms, specifically when considering factors such as student motivation and the exhaustion experienced in this regard.

Data Availability Statement

The datasets presented in this study can be found in online repositories. This data can be found here: https://data.mendeley.com/datasets/p5d39f3kp6/1.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Committee of Universidad del Desarrollo, Santiago de Chile (protocol number: 2020–142). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

MR-R, AJC, and DF-O contributed to the conception, coordination, and design of the work. MR-R and DF-O contributed to the implementation of the program. MR-R and AJC performed the statistical analysis. All authors contributed to the article and approved the submitted version.

Funding

This work was funded by the Spanish Ministry of Science, Innovation and Universities: (PSI2017-84961-R AEI/FEDER, UE), and by the National Agency for Research and Development (ANID)/Scholarship Program/MAGISTER NACIONAL/2020–22200859.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2021.632252/full#supplementary-material

References

1. WHO-AIMS. Evaluación del Sistema de Salud Mental de Chile: Segundo inform. Informe Anual Sobre Derechos Humanos En Chile 2014. Ginebra: WHO (2014). Available online at: https://www.who.int/mental_health/who_aims_country_reports/who_aims_report_chile.pdf

2. Thornicroft G, Rose D, Kassam A, Sartorius N. Stigma: Ignorance, prejudice or discrimination? Br J Psychiatry. (2007) 190:192–3. doi: 10.1192/bjp.bp.106.025791

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Corrigan PW, Shapiro JR. Measuring the impact of programs that challenge the public stigma of mental illness. Clin Psychol Rev. (2010) 30:907–22. doi: 10.1016/j.cpr.2010.06.004

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Mascayano F, Lips W, Mena C, Manchego C. Estigma hacia los trastornos mentales: características e intervenciones. Salud Ment. (2015) 38:53–8. Available online at: http://www.scielo.org.mx/scielo.php?script=sci_arttext&pid=S0185-33252015000100008&lng=es&tlng=es (accessed November 11, 2020).

Google Scholar

5. López M. Work and employment for people with severe mental disorders. Some proposals based on the available information. Revi Asoc Esp Neuropsiq. (2004) 24:31–66. Available online at: http://dialnet.unirioja.es/servlet/articulo?codigo=864102&info=resumen&idioma=SP (accessed November 11, 2020)

6. Yang L, Valencia E, Alvarado R, Link B, Huynh N, Nguyen K, et al. A theoretical and empirical framework for constructing culture-specific stigma instruments for Chile. Cad Saude Colet. (2013) 21:71–9. doi: 10.1590/S1414-462X2013000100011

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Vicente B, Saldivia S, Pihán R. Prevalencias y brechas hoy; salud mental mañana. Acta Bioeth. (2016) 22:51–61. Available online at: https://scielo.conicyt.cl/scielo.php?script=sci_arttext&pid=S1726-569X2016000100006 (accessed November 11, 2020)

Google Scholar

8. Navarro N, Triguero R. Estigma en los profesionales de la Salud Mental: una revisión sistemática. Psychol Soc Educ. (2019) 11:253–66. doi: 10.25115/psye.v11i2.2268

CrossRef Full Text | Google Scholar

9. Arens E, Berger C, Lincoln T. Stigmatization of patients with schizophrenia: the influence of university courses on the attitudes of prospective psychologists and doctors. Nervenarzt. (2009) 80:329–39. doi: 10.1007/s00115-008-2646-4

PubMed Abstract | CrossRef Full Text

10. Funk M, Drew N, Knapp M. Mental health, poverty and development. JPMH. (2012) 11:166–85. doi: 10.1108/17465721211289356

CrossRef Full Text | Google Scholar

11. Hernández de Cadena N, Escobar C. Una aproximación al maltrato a los pacientes con trastornos mentales. MedUNAB. (2004) 7:3. Available online at: http://revistas.unab.edu.co/index.php?journal=medunab&page=article&op=viewFile&path%5B%5D=231&path%5B%5D=214 (accessed November 11, 2020)

PubMed Abstract | Google Scholar

12. Khenti A, Bobbili S, Sapag J. Evaluation of a pilot intervention to reduce mental health and addiction stigma in primary care settings. J Community Health. (2019) 44:1204–13. doi: 10.1007/s10900-019-00706-w

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Poreddi V, Thimmaiah R, BadaMath S. Medical and nursing students' attitudes toward mental illness: an Indian perspective. Invest Educ Enferm. (2017) 35:86–94. Available online at: https://doi.org/10.17533/udea.iee.v35n1a10 (accessed November 11, 2020)

PubMed Abstract | Google Scholar

14. Jadhav S, Littlewood R, Ryder A, Chakraborty A, Jain S, Barua M. Stigmatization of severe mental illness in India: against the simple industrialization hypothesis. Indian J Psychiatry. (2007) 43:189–94. doi: 10.4103/0019-5545.37320

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Vidourek R, Burbage M. Positive mental health and mental health stigma: A qualitative study assessing student attitudes. MHP. (2019) 13:1–6. doi: 10.1016/j.mhp.2018.11.006

CrossRef Full Text | Google Scholar

16. Yin H, Wardenaar K, Xu G, Tian H, Schoevers R. Mental health stigma and mental health knowledge in Chinese population: a cross-sectional study. BMC Psychiatry. (2020) 20:323. doi: 10.1186/s12888-020-02705-x

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Lien Y, Lin H, Tsai C, Lien Y, Wu T. Changes in attitudes toward mental illness in healthcare professionals and students. Int J Environ Res Public Health. (2019) 16:4655. doi: 10.3390/ijerph16234655

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Schulze B. Stigma and mental health professionals: a review of the evidence on an intricate relationship. Int Rev Psychiatry. (2007) 19:137–55. doi: 10.1080/09540260701278929

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Thornicroft G, Mehta N, Clement S, Evans-Lacko S, Doherty M, Rose D, et al. Evidence for effective interventions to reduce mental-health-related stigma and discrimination. Lancet. (2016) 387:1123–32. doi: 10.1016/s0140-6736(15)00298-6

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Corrigan P, Wassel A. Understanding and influencing the stigma of mental illness. J Psychosoc Nurs Ment Health Serv. (2008) 46:42–8. doi: 10.3928/02793695-20080101-04

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Yuan Q, Picco L, Chang S, Abdin E, Cuha B, Ong S, et al. Attitudes to mental illness among mental health professionals in Singapore and comparisons with the general population. PLoS One. (2017). doi: 10.1371/journal.pone.0187593

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Kadri N, Sartorius N. The global fight against the stigma of schizophrenia. PLoS Med. (2005) 2:e136. doi: 10.1371/journal.pmed.0020136

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Knaak S, Modgill G, Patten S. Key ingredients of anti-stigma programs for health care providers: a data synthesis of evaluative studies. Can J Psychiatry. (2014) 59:19–26. doi: 10.1177/070674371405901s06

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Lanfredi M, Macis A, Ferrari C, Rillosi L, Cadone E, Fanetti A, et al. Effects of education and social contact on mental health-related stigma among high-school students. Psychiatry Res. (2019) 281:112581. doi: 10.1016/j.psychres.2019.112581

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Corrigan PW, Michaels PJ, Vega E, Gause M, Larson J, Krzyzanowski R, et al. Key ingredients to contact-based stigma change: A cross-validation. Psychiatr Rehabil J. (2014) 37:62–4. doi: 10.1037/prj0000038

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Corrigan PW, Vega E, Larson J, Michaels PJ, McClintock G, Krzyzanowski R, et al. The California schedule of key ingredients for contact-based antistigma programs. Psychiatr Rehabil J. (2013) 36:173–9. doi: 10.1037/prj0000006 (accessed January 8, 2021).

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Morgan A, Reavley N, Ross A, Too LS, Jorm A. Interventions to reduce stigma towards people with severe mental illness: Systematic review and meta-analysis. J Psychiatr Res. (2018) 103:120–33. doi: 10.1016/j.jpsychires.2018.05.017

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Hickie I. Can we reduce the burden of depression? The Australian experience with beyondblue: the national depression initiative. Australas Psychiatry. (2004) 12:S38–S46. doi: 10.1111/j.1039-8562.2004.02097.x

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Ministry of Health and Health Promotion Agency. Like Minds, Like Mine National Plan 2014-2019: Programme to Increase Social Inclusion and Reduce Stigma and Discrimination for People with Experience of Mental Illness. Wellington: Ministry of Health (2014).

Google Scholar

30. Stuart H, Chen S, Christie R, Dobson K, Kirsh B, Knaak S, et al. Opening minds in canada: targeting change. Can J Psychiatry. (2014) 59:13–8. doi: 10.1177/070674371405901s05

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Corrigan P, Rafacz J, Hautamaki J, Walton J, Rüsch N, Rao D, et al. Changing stigmatizing perceptions and recollections about mental illness: the effects of NAMI's in our own voice. Community Ment Health J. (2010) 46:517–22. doi: 10.1007/s10597-009-9287-3

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Pelzang R. Attitude of nurses towards mental illness in Bhutan. J Bhutan Studies. (2010) 22:60–76. Available online at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.731.1095&rep=rep1&type=pdf (accessed November 11, 2020)

Google Scholar

33. Corrigan P. Lessons learned from unintended consequences about erasing the stigma of mental illness. World Psychiatry. (2016) 15:67–73. doi: 10.1002/wps.20295

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Lam A, Jorm A, Wong D. Mental health first aid training for the Chinese community in Melbourne, Australia: Effects on knowledge about and attitudes toward people with mental illness. IJMHS. (2010) 4:18. doi: 10.1186/1752-4458-4-18

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Shann C, Martin A, Chester A, Ruddock S. Effectiveness and application of an online leadership intervention to promote mental health and reduce depression-related stigma in organizations. J Occup Health Psychol. (2019) 24:20–35. doi: 10.1037/ocp0000110

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Amsalem D, Yang L, Jankowski S, Lieff S, Markowitz J, Dixon L. Reducing stigma toward individuals with schizophrenia using a brief video: a randomized controlled trial of young adults. Schizophr Bull. (2020) 2020:sbaa114. doi: 10.1093/schbul/sbaa114

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Amsalem D, Gothelf D, Soul O, Dorman A, Ziv A, Gross R. Single-day simulation-based training improves communication and psychiatric skills of medical students. Front Psychiatry. (2020) 11:221. doi: 10.3389/fpsyt.2020.00221

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Maunder RD, White FA, Verrelli S. Modern avenues for intergroup contact: using E-contact and intergroup emotions to reduce stereotyping and social distancing against people with schizophrenia. GPIR. (2019) 22:947–63. doi: 10.1177/1368430218794873

CrossRef Full Text | Google Scholar

39. Cangas AJ, Navarro N, Aguilar JM, Trigueros R, Gallego J, Zárate R, et al. Analysis of the usefulness of a serious game to raise awareness about mental health problems in a sample of high school and university students: relationship with familiarity and time spent playing video games. J Clin Med. (2019) 8:1504. doi: 10.3390/jcm8101504

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Mullor D, Sayans P, Cangas A, Navarro N. Effect of a Serious Game (Stigma-Stop) on reducing stigma among psychology students: a controlled study. Cyberpsychol Behav Soc Netw. (2019) 22:205–11. doi: 10.1089/cyber.2018.0172

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Javed B, Sarwer A, Soto E, Mashwani Z. Impact of SARS-CoV-2 (coronavirus) pandemic on public mental health. Public Health Front. (2020) 8:292. doi: 10.3389/fpubh.2020.00292

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Yao H, Chen J, Xu Y. Patients with mental health disorders in the COVID-19 epidemic. Lancet Psychiatry. (2020) 7:E21. doi: 10.1016/S2215-0366(20)30090-0

PubMed Abstract | CrossRef Full Text | Google Scholar

43. King B, Smith C. Using project-based learning to develop teachers for leadership. J Educ Strat Issues Ideas. (2020) 93:158–64. doi: 10.1080/00098655.2020.1735289

CrossRef Full Text | Google Scholar

44. MacLeod M, van der Veen J. Scaffolding interdisciplinary project-based learning: a case study. Eur J Eng Educ. (2020) 45:363-77. doi: 10.1080/03043797.2019.1646210

CrossRef Full Text | Google Scholar

45. Wang J, Tsiligiris V, Hartley R. Perceptions of the benefits of the A3 planner in facilitating project-based learning in accounting education. JAEd. (2019) 28:221–38. doi: 10.1080/09639284.2018.1516151

CrossRef Full Text | Google Scholar

46. Bell B. Project-Based Learning for the 21st Century: Skills for the Future. J Educ Strat Issues Ideas. (2010) 83:39–43. doi: 10.1080/00098650903505415

CrossRef Full Text | Google Scholar

47. Burke B, Currin-McCulloch J. Educating our future colleagues: creating an MSW palliative care practicum. J Soc Work End Life Palliat Care. (2019) 15:7–15. doi: 10.1080/15524256.2019.1586806

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Leggett G, Harrington I. The impact of Project Based Learning (PBL) on students from low socio economic statuses: a review. Int J Incl Educ. (2019) 1–17. doi: 10.1080/13603116.2019.1609101

CrossRef Full Text | Google Scholar

49. Chen CH, Yang YC. Revisiting the effects of project-based learning on students' academic achievement: a meta-analysis investigating moderators. Educ Res Rev. (2019) 26:71–81. doi: 10.1016/j.edurev.2018.11.001

CrossRef Full Text | Google Scholar

50. Hunt J, Brannan E, Sexson S. Lifelong learning for professional development in psychiatry: pedagogy, innovations, and maintenance of certification. Psychiatr Clin N. (2019) 42:425–37. doi: 10.1016/j.psc.2019.05.014

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Bokken L, Rethans JJ, Jöbsis Q, Duvivier R, Scherpbier A, van der Vleuten C. Instructiveness of real patients and simulated patients in undergraduate medical education: a randomized experiment. Acad Med. (2010) 85:148–54. doi: 10.1097/ACM.0b013e3181c48130

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Karim A, Campbell M, Hasan M. A new method of integrating project-based and work-integrated learning in postgraduate engineering study. Curric J. (2019) 31:157–73. doi: 10.1080/09585176.2019.1659839

CrossRef Full Text | Google Scholar

53. Schomerus G, Angermeyer MC, Baumeister SE, Stolzenburg S, Link BG, Phelan JC. An online intervention using information on the mental health-mental illness continuum to reduce stigma. Eur Psychiatry. (2016) 32:21–7. doi: 10.1016/j.eurpsy.2015.11.006

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Martin A, Krause R, Jacobs A, Chilton J, Amsalem D. The mental status exam through video clips of simulated psychiatric patients: an online educational resource. Acad Psychiatry. (2020) 44:179–83. doi: 10.1007/s40596-019-01140-9

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Tippin GK, Maranzan KA. Efficacy of a Photovoice-based video as an online mental illness anti-stigma intervention and the role of empathy in audience response: A randomized controlled trial. J Appl Soc Psychol. (2019) 49:381–94. doi: 10.1111/jasp.12590

CrossRef Full Text | Google Scholar

56. Martin A, Krause R, Chilton J, Jacobs A, Amsalem D. Attitudes to psychiatry and to mental illness among nursing students: adaptation and use of two validated instruments in preclinical education. J Psychiatr Ment Health Nurs. (2020) 27:308–17. doi: 10.1111/jpm.12580

PubMed Abstract | CrossRef Full Text | Google Scholar

57. White F, Abu-Rayya H. A dual identity-electronic contact (DIEC) experiment promoting short-and long-term inter-group harmony. J Exp Soc Psychol. (2012) 48:597–608. doi: 10.1016/j.jesp.2012.01.007

CrossRef Full Text | Google Scholar

58. Hodson G. Do ideologically intolerant people benefit from intergroup contact? Curr Dir Psychol. (2011) 20:154–9. doi: 10.1177/0963721411409025

CrossRef Full Text | Google Scholar

59. White F, Turner R, Verrelli S, Harvey L, Hanna J. Improving intergroup relations between catholics and protestants in Northern Ireland via E-contact. Eur J Soc Psychol. (2019) 49:429–38. doi: 10.1002/ejsp.2515

CrossRef Full Text | Google Scholar

60. Boccanfuso E, White FA, Maunder RD. Reducing transgender stigma via an E-contact intervention. Sex Roles. (2020) 1–11. doi: 10.1007/s11199-020-01171-9

CrossRef Full Text | Google Scholar

61. Martin A, Chilton J, Paasche C, Nabatkhorian N, Gortler H, Cohenmehr E, et al. Shared living experiences by physicians have a positive impact on mental health attitudes and stigma among medical students: a mixed-methods study. J Med Educ Curric Dev. (2020) 7:1–9. doi: 10.1177/2382120520968072

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Schulze B, Richter-Werling M, Matschinger H, Angermeyer M.C. Crazy? So what! Effects of a school project on students' attitudes toward people with schizophrenia. Acta Psychiatr Scand. (2003) 107:142–50. doi: 10.1034/j.1600-0447.2003.02444.x

CrossRef Full Text | Google Scholar

63. Navarro N, Cangas A, Aguilar-Parra J, Gallego J, Moreno-San Pedro E, Carrasco-Rodríguez Y, et al. Propiedades psicométricas de la versión en castellano del cuestionario de las actitudes de los estudiantes hacia la esquizofrenia. Psychol Soc Educ. (2017) 9:325–34. doi: 10.25115/psye.v9i2.865

CrossRef Full Text | Google Scholar

64. Corrigan P, Markowitz F, Watson A, Rowan D, Kubiak M. An attribution model of public discrimination towards persons with mental illness. J Health Soc Behav. (2003) 44:162–79. doi: 10.2307/1519806

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Saavedra J, Arias-Sánchez S, Corrigan P, López M. Assessing the factorial structure of the mental illness public stigma in Spain. Disabil Rehabil. (2020) 6:1–7. doi: 10.1080/09638288.2019.1710769

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Asociación Médica Mundial. Declaración de Helsinki de la AMM – Principios Éticos Para las Investigaciones Médicas en Seres Humanos. (2017). Available online at: https://www.wma.net/es/policies-post/declaracion-de-helsinki-de-la-amm-principios-eticos-para-las-investigaciones-medicas-en-seres-humanos/ (accessed November 11, 2020).

Google Scholar

67. Resnik D, Shamoo A. The Singapore statement on research integrity. Account Res. (2011) 18:71–5. doi: 10.1080/08989621.2011.557296

CrossRef Full Text | Google Scholar

68. Howells FM, Kingdon DG, Baldwin DS. Current and potential pharmacological and psychosocial interventions for anxiety symptoms and disorders in patients with schizophrenia: structured review. Hum Psychopharmacol. (2017) 32:5. doi: 10.1002/hup.2628

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Khokhar JY, Dwiel LL, Henricks AM, Doucette WT, Green AI. The link between schizophrenia and substance use disorder: a unifying hypothesis. Schizophr Res. (2018) 194:78–85. doi: 10.1016/j.schres.2017.04.016

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Wahl O, Aroesty-Cohen E. Attitudes of mental health professionals about mental illness: a review of the recent literature. J Community Psychol. (2010) 38:49–62. doi: 10.1002/jcop.20351

CrossRef Full Text | Google Scholar

71. Stefanovics E, Rosenheck R, He H, Ofori-Atta A, Cavalcanti M, Chiles C. Medical student belief and attitudes toward mental illness across five nations. J Nerv Ment Dis. (2016) 204:909–15. doi: 10.1097/NMD.0000000000000610

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Gallego J, Cangas A, Aguilar J, Trigueros R, Navarro N, Galván B, et al. Education students' stigma toward mental health problems: a cross-cultural comparison. Front Psychiatry. (2020) 11:587321. doi: 10.3389/fpsyt.2020.587321

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Kemp C, Jarrett B, Kwon C, Song L, Jetté N, Sapag J, et al. Implementation science and stigma reduction interventions in low- and middle-income countries: a systematic review. BMC Med. (2019) 17:6. doi: 10.1186/s12916-018-1237-x

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Maunder RD, White F. Intergroup contact and mental health stigma: A comparative effectiveness meta-analysis. Clin Psychol Rev. (2019) 72:1749. doi: 10.1016/j.cpr.2019.101749

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Kitay B, Martin A, Chilton J, Amsalem D, Duvivier R, Goldenberg M. Electroconvulsive therapy: a video-based educational resource using standardized patients. Acad Psychiatry. (2020) 44:531–7 doi: 10.1007/s40596-020-01292-z

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Nickerson A, Byrow Y, Pajak R, McMahon T, Bryant R, Christensen H, et al. ‘Tell Your Story’: A randomized controlled trial of an online intervention to reduce mental health stigma and increase help-seeking in refugee men with posttraumatic stress. Psychol Med. (2020) 50:781–92. doi: 10.1017/S0033291719000606

PubMed Abstract | CrossRef Full Text | Google Scholar

77. Inan FS, Günüşen N, Duman ZÇ, Ertem MY. The impact of mental health nursing module, clinical practice and an anti-stigma program on nursing students' attitudes toward mental illness: a quasi-experimental study. J Prof Nurs. (2019) 35:201–8. doi: 10.1016/j.profnurs.2018.10.001

PubMed Abstract | CrossRef Full Text | Google Scholar

78. Koike S, Yamaguchi S, Ojio Y, Ohta K, Shimada T, Watanabe K, et al. A randomised controlled trial of repeated filmed social contact on reducing mental illness-related stigma in young adults. Epidemiol Psychiatr Sci. (2018) 27:199–208. doi: 10.1017/S2045796016001050

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Dedeilia A, Sotiropoulos MG, Hanrahan JG, Janga D, Dedeilias P, Sideris M. Medical and surgical education challenges and innovations in the COVID-19 era: a systematic review. In Vivo. (2020) 34(3 Suppl):1603–11. doi: 10.21873/invivo.11950

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Chick RC, Clifton GT, Peace KM, Propper BW, Hale DF, Alseidi AA, et al. Using technology to maintain the education of residents during the COVID-19 pandemic. J Surg Educ. (2020) 77:729–32. doi: 10.1016/j.jsurg.2020.03.018

PubMed Abstract | CrossRef Full Text | Google Scholar

81. Tabatabai S. COVID-19 impact and virtual medical education. J Adv Med Educ Prof. (2020) 8:140–3. doi: 10.30476/jamp.2020.86070.1213

CrossRef Full Text | Google Scholar

82. Sabharwal S, Ficke JR, LaPorte DM. How we do it: modified residency programming and adoption of remote didactic curriculum during the COVID-19 pandemic. J Surg Educ. (2020) 77:1033–6. doi: 10.1016/j.jsurg.2020.05.026

PubMed Abstract | CrossRef Full Text | Google Scholar

83. Amsalem D, Gothelf D, Dorman A, Goren Y, Tene O, Shelef A, et al. Reducing stigma toward psychiatry among medical students: a multicenter controlled trial. Prim Care Companion CNS Disord. (2020) 22:19m02527. doi: 10.4088/PCC.19m02527

PubMed Abstract | CrossRef Full Text | Google Scholar

84. López M, Laviana M, Fernández L, López A, Rodríguez AM, Aparicio A. La lucha contra el estigma y la discriminación en salud mental. Una estrategia compleja basada en la información disponible. Rev Asoci Esp Neuropsiquiatr. (2008) 28:48–83. doi: 10.4321/S0211-57352008000100004

CrossRef Full Text | Google Scholar