Psychometric properties of the Opening Minds Stigma Scale for Health Care Providers in 32 European countries – A bifactor ESEM representation

Aims: To measure the stigma of healthcare providers toward people suffering from mental illness, the Opening Minds Stigma Scale for Health Care Providers (OMS-HC) is a commonly applied instrument. However, this scale has not been thoroughly validated in many European countries, its psychometric properties are still unknown and data on practicing psychiatrists is lacking. Therefore, this multicenter study aimed to assess the psychometric characteristics of the 15-item OMS-HC in trainees and specialists in adult and child psychiatry in 32 countries across Europe.

Materials and methods: The OMS-HC was conducted as an anonymous online survey and sent via Email to European adult and child psychiatrists. Parallel analysis was used to estimate the number of OMS-HC dimensions. Separate for each country, the bifactor ESEM, a bifactor exploratory structural equation modeling approach, was applied to investigate the factor structure of the scale. Cross-cultural validation was done based on multigroup confirmatory factor analyses and reliability measures.

Results: A total of 4,245 practitioners were included, 2,826 (67%) female, 1,389 (33%) male. The majority (66%) of participants were specialists, with 78% working in adult psychiatry. When country data were analyzed separately, the bifactor model (higher-order factor solution with a general factor and three specific factors) showed the best model fit (for the total sample χ²/df = 9.760, RMSEA = 0.045 (0.042–0.049), CFI = 0.981; TLI = 0.960, WRMR = 1.200). The average proportion of variance explained by the general factor was high (ECV = 0.682). This suggests that the aspects of ‘attitude,’ ‘disclosure and help-seeking,’ and ‘social distance’ could be treated as a single dimension of stigma. Among the specific factors, the ‘disclosure and help-seeking’ factor explained a considerable unique proportion of variance in the observed scores.

Conclusion: This international study has led to cross-cultural analysis of the OMS-HC on a large sample of practicing psychiatrists. The bifactor structure displayed the best overall model fit in each country. Rather than using the subscales, we recommend the total score to quantify the overall stigmatizing attitudes. Further studies are required to strengthen our findings in countries where the proposed model was found to be weak.

1. Introduction

People with mental illness are frequently the targets of stigma and discrimination, which can have serious repercussions for both the stigmatized individuals and the larger society. Unfortunately, stigma affects people with mental illness not only in their daily lives but also when they seek medical care, including mental care, which results in poor physical and mental health of the person (1, 2). The experienced stigma from the service staff severely impacts the life expectancy and quality of life of individuals and serves as a predictor of their internalized stigma and disempowerment (3).

Developed to evaluate anti-stigma programs for healthcare workers, the Opening Minds Stigma Scale for Health Care Providers (OMS-HC) is one of the most widely used self-report measurements of stigmatizing attitudes worldwide. The authors of the scale created a pool of 50 items that they collected from existing scales and developed new items by consulting people who work in anti-stigma programs for healthcare providers. The pool was reviewed by professionals and people with lived experience, which was followed by cognitive interviewing of healthcare workers, and seven focus groups were held to improve the wording of the scale. Finally, the changes resulted in a reduction to 20 items (4). The factor analysis of the data resulted in a two-factor solution (attitude and disclosure) with 12 items that missed the important dimension of social distance. For this reason, the authors revisited the factor structure of the scale in a larger and more representative sample consisting of various healthcare providers that resulted in a more stable, three-factor solution (attitude, disclosure and help-seeking, and social distance) with 15 items (5). Studies with different samples of healthcare providers have shown OMS-HC scores to be a reliable and valid measure of stigmatizing attitudes toward patients in Italy (6), Hungary (7), Germany and Switzerland (8), Chile (9), and Singapore (10). The three dimensions of attitude, disclosure and help-seeking, and social distance structure appear to be similar; however, in most of the international studies, some items were found to be weak and loaded on different factors than that of the Canadian version and theory. Table 1 summarizes the factor analysis results on the 15-item OMS-HC in international studies.

TABLE 1

Table 1. Overview of factor analyses results on the 15-item OMS-HC in international studies.

Participant samples of factor analytic studies included professionally heterogeneous samples, in which mental healthcare samples were composed predominantly of mental health nurses, medical students, and psychologists. Two psychometric studies on the scale included psychiatrists (subsample sizes in Germany and Switzerland together n = 49, Canada n = 79), and solely one study examined the factor structure of the scale on a psychiatrist sample (Hungary n = 211). Therefore, it is unknown whether the extent of findings could be generalized to the psychiatrist population, which is critical because investigating their stigmatizing attitudes toward people with mental health problems is considered an important and understudied area of research. Based on the current literature, the results are mixed; in some cases, their attitudes have been found to be more favorable, whereas in others they expressed more stigma than that of other healthcare providers (12–14). To ensure the cross-cultural applicability of the scale in evaluating the attitudes of psychiatrists, we aimed to choose a sample as homogeneous as possible; thus, the indicators of the instrument are not affected by characteristics of different occupations. The psychiatric community needs to have a measurement tested on a sample that consists solely of psychiatrists. It will allow using the OMS-HC in effectiveness and other measures of anti-stigma interventions among them, and the results can be interpreted in a cultural context. There is also a lack of multicenter studies in the literature on the attitudes of psychiatrists. However, to be able to perform cross-cultural measurements, the construct represented by the questionnaire needs to be simultaneously tested and confirmed by using different language versions.

Thus far, Chang in Singapore (10) and Sapag in Chile (9), have analyzed the psychometric properties of the OMS-HC by using exploratory structural equation modeling(ESEM). This method provides us with a suitable way to analyze attitude measurements because it combines the benefits of more sophisticated confirmatory factor analysis (such as goodness-of-fit or multigroup models) with less constrained exploratory factor analysis. Moreover, as opposed to confirmatory factor analysis, ESEM is expected to relieve more accurate correlations between latent factors, enhancing their discriminant validity, and leading to a more realistic representation of the data (15). The bifactor ESEM approach is a hierarchically ordered construct that provides a clear and explicit estimate of the overall stigmatizing attitude (general factor) along with specific factors that represent the individual features of each subscale that are not explained by the general factor (16). By allowing cross-loadings to be freely assessed among all elements used to reflect stigmatizing attitudes, this framework also considers the fallible nature of the indicators used to evaluate each construct (17).

No studies have scoped the factor structure of the OMS-HC through bifactor ESEM, and the OMS-HC has not yet been investigated across numerous psychiatrist samples from different countries. For these reasons, it is essential to know whether the OMS-HC scores measure the same construct in the same manner across psychiatrists from different countries with various cultural and ethnic backgrounds.

Based on the scientific literature, we aimed to test whether the three correlated factors or the bifactor ESEM approach to the OMS-HC is more appropriate in a diverse sample of practicing psychiatrists from 32 European countries. As an exploratory second aim, our study investigated the model-based reliability of the bifactor structure.

2. Materials and methods

2.1. Study design

In this cross-sectional research study, an anonymous online self-reported survey was distributed directly among adult and child psychiatrists in the participating 32 European countries. In each country, a dedicated psychiatrist investigator submitted the study protocol for review according to the local regulations, arranged the translation of the survey questions, and enrolled participants. In countries where the OMS-HC had not been available, it was translated according to the following guidelines: The English version was first translated into the local language by two different colleagues in psychiatry with a good command of English, then the two versions were reconciled into a single version, and then an independent translator back-translated it to English. An iterative procedure was used to resolve any discrepancies between the original and the back-translated versions of the scale; then, the final local version was sent to the participants. The survey was modified in case it was needed based on the comment of the first respondents. In countries with more than one official language, multilingual surveys (French, German, Italian in Switzerland, and Flemish and French in Belgium) were used to approach a broader population. Investigators distributed the questionnaire by contacting the national psychiatric associations, universities, local hospitals, and health centers and through professional networks, including social media platforms and professional groups’ email lists.

2.2. Participants

The combined total sample (n = 4,245) consisted of trainees and specialists in general adult and child psychiatry from 32 European countries. Not working in psychiatric care and medical student status were the exclusion criteria.

2.3. Measures

All participants completed the OMS-HC, and basic sociodemographic information (age range, gender, status, and place of work) was also gathered from them. The 15-item long OMS-HC is a self-report scale, which is conceptualized as an attitude measure comprising three components (attitude, disclosure and help-seeking, and social distance). Specifically, attitude is measured by six items (e.g., ‘I struggle to feel compassion for a person with a mental illness.’), disclosure and help-seeking, with four items (e.g., ‘If I had a mental illness, I would tell my friends.’), and social distance is measured by five items (e.g., ‘I would not mind if a person with a mental illness lived next door to me.’). Subjects indicate on a 5-point Likert scale the extent to which they identify themselves with the given statements from ‘strongly disagree’ to ‘strongly agree.’ The total scores range from 15 to 75, and the subscales from 6 to 30, 4 to 20, and 5 to 25, respectively. Higher scores reflect a more stigmatizing attitude. Before this study, the psychometric properties of the OMS-HC were investigated in Italy, Hungary, Germany, and Switzerland (German part) in the participating countries, and it was found to be a reliable and valid measurement of the stigmatizing attitudes (6–8).

2.4. Statistical analyses

The percentage (%) and sample size (n) are used to express the demographic data. First, using parallel analysis to compare the progressive eigenvalues from the given data matrix to those of a simulated data matrix created using random data of the same size, we determined the number of elements to be extracted (18). The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was calculated to ensure that the matrices were suitable for exploratory factor (EFA) (should be >0.60). Bartlett’s Test of Sphericity was used to ensure the non-randomness of the correlation matrix (value of p should be 0.05). All confirmatory factor analyses and ESEM models used diagonally weighted least squares (WLSMV) to find the optimum model for each country. It accounts for standard errors and non-normality in chi-square test statistics for ordinal data. We tested unidimensional, three-factor, and bifactor model fit. We used the following indices to evaluate model fit: chi-square / degree of freedom (χ²/df) (<5.0), root mean square error of approximation (RMSEA, <0.06 as good fit, 0.06–0.08 as moderate fit, 0.08–0.1 as marginal fit, > 0.1 as poor fit), comparative fit index (CFI) and Tucker-Lewis Index (TLI), CFI and TLI (> 0.95 as good fit, >0.90 as acceptable, and < 0.90 as poor fit) (19). Model-based reliability was assessed by omega hierarchical (ωH), explained common variance (ECV), and percent uncontaminated correlations (PUC). These show that total and subscale scores accurately represent the target constructs. The ω (values range 0–1, with 0 indicating no reliability and 1 reflecting perfect reliability) can be used to estimate the factor’s contribution to summed (standardized) scores (20). Reise advised that the ωH should be greater than 0.50 and ideally greater than 0.75 (16). When PUC values are above 0.80, general ECV values predict bias less well. If the PUC values are less than 0.80, the general ECV values are larger than 0.60, and the ωH values are greater than 0.70, the instrument can be interpreted as unidimensional (16). Spearman’s correlation analyzed the associations between specific and general factors. IBM SPSS 26.0.0.0 (21) and MPlus Version 8.8 (22) statistical software were used for analysis.

3. Results

A total of n = 4,245 professionals in psychiatry completed the survey. The majority of the participants were female (n = 2,856, 67%), while young professionals between the ages of 24 and 35 years (n = 1856, 44%) were overrepresented in the sample. Most of them worked in adult psychiatry (n = 3,320, 78%) and inpatient facilities (n = 1884, 44%). Working facility of the participants and their years of experience in psychiatry are attached as a Supplementary material S1. There were no missing values in the dataset.

First, based on the suitable Bartlett’s Test of Sphericity (p < 0.0001) and the KMO measure of sampling adequacy (0.841) we performed an EFA on the pooled sample to see the factor matrix, the network of the items, and to create an overview of the structure of the scale. To determine the number of factors for extraction, parallel analysis was performed, and the eigenvalues were considered, which identified 3 or 4 different dimensions. As all item loadings were lower on the fourth factor than the other three, therefore, we continued with the three-factor structure that accounted for a sufficient amount of variance (25.1, 9.9, and 8.54%, respectively).

EFA results are found in the Supplementary material S2.

A series of confirmatory factor analyses were performed to find the best model fit for the OMS-HC in each participating country. The bifactor ESEM model was found to be superior to the alternative models by showing an acceptable fitting level. This model (see Figure 1 for a comparison with the correlated model) consists of a general factor (loads directly onto all items) and three specific factors (load onto sub-groups of the same set of observed variables). The fit measures and the average loadings for the bifactor model are shown in Table 2. As presented, the model had excellent or acceptable levels of fit in most countries (29/32 countries). Across the 32 countries, the RMSEA ranged from 0.000 to 0.131 (M = 0.05, SD = 0.03), CFI ranged from 0.870 to 1.000 (M = 0.97, SD = 0.03), TLI ranged from 0.732 to 1,084 (M = 0.94, SD = 0.06), and the WRMR ranged from 0.304 to 0.651 (M = 0.47, SD = 0.07). However, the model failed to run in Albania and Swiss French and Swiss Italian languages due to the negative covariance matrices (theta). The absolute index, RMSEA, was moderate in Bulgaria, Cyprus, Greece, Italy, Latvia and Turkey, and it was associated with slightly lower than acceptable relative fit indices, CFI and/or TLI in Azerbaijan, Latvia, Montenegro, and Slovakia. The fit indices for the total sample were within the recommended ranges. Alternatively, the unidimensional and the correlated three-factor models were considered. The Supplementary material S3 contains their fit indices, which were poor and the models failed to run in several countries. In the unidimensional model, the RMSEA ranged from 0.078 to 0.172 (M = 0.12, SD = 0.02), the CFI ranged from 0.579 to 0.902 (M = 0.78, SD = 0.09), and TLI ranged from 0.509 to 0.886 (M = 0.75, SD = 0.10). The three-factor model showed a little better fit, with RMSEA ranging from 0.054 to 0.124 (M = 0.08, SD = 0.02), CFI and TLI ranging from 0.726 to 0.966 (M = 0.89, SD = 0.06) and 0.669 to 0.959 (M = 0.86, SD = 0.07), respectively. All fit indices indicate that the bifactor ESEM model provides us with the best model fit.

FIGURE 1

Figure 1. Simplified conceptual representations of the estimated models, ESEM cross-loadings (in the ESEM models all items are allowed to cross-load on all the specific-factors) are excluded in this figure to avoid cluttering.

TABLE 2

Table 2. Results of the bifactor ESEM model in each participating country.

The specific and general factor of the OMS-HC were strongly correlated: Attitude: r = 0.791, p < 0.0001; Disclosure: r = 0.680, p < 0.0001; Social distance: r = 0.785. Additionally, the specific factors correlated statistically significantly with each other: Attitude and Disclosure: r = 0.294, p < 0.001; Attitude and Social distance: r = 0.506, p < 0.0001; Disclosure and Social distance: r = 0.310, p < 0.0001.

In the next step, we tested the model-based reliability (see Table 3). The general factor indicated acceptable model-based reliability for the whole sample. However, the reliability of the specific factors was found to be poor. Among the specific factors, the Disclosure and Help-seeking was the one that approached the limit of acceptable reliability. As the PUC value for the model was 0.705 and ECV and ωH of the general factor were 0.682 and 0.746, respectively.

TABLE 3

Table 3. Estimates of model-based reliability of the general and specific factors.

4. Discussion

The primary aim of this study was to examine the factor structure of the OMS-HC in 32 countries across Europe on a large group of psychiatrists. To achieve our goal, after gaining an insight into the dimensionality of the pooled sample by parallel analysis, we performed a CFA series and compared the fit indices of the unidimensional, the three-correlated, and the bifactor ESEM models for each participating country. We specifically examined whether the proposed bifactor ESEM model of the OMS-HC is replicable across cultures and whether it more accurately captures the factor structure in comparison to competing models, especially the three-factor model proposed by Modgill (5), which was replicated in Italy, Chile, Singapore and on a German-Swiss sample (6, 8–10).

Both the parallel analysis and the EFA, strongly supported the 3-dimensionality of the scale on the pooled sample. Therefore, it has been our starting point for investigating the structure. The simple three-dimensional structure would have been in line with the Canadian, Italian, Chilean, and German-Swiss results. However, note that these studies detected significant cross-loadings of items between factors. Moreover, one item was removed from the analyses in Singapore due to poor loading across all three factors.

Because of the inherent limitations of the simple three-factor structure, we decided to investigate the model further. Our research has demonstrated that the bifactor model provided us with the best approximation of the factor structure of the OMS-HC with a good to acceptable fit in nearly all countries. It seemed superior to alternative models, including the three-factor correlated trait model based on the fit indices. In fact, our model does not contradict the three-factor model; furthermore, it investigates it for a higher-order solution. The three-dimensions are identified as three specific factors, supplemented by a general factor in a hierarchical structure resulting in better model fit. Although it showed a good fit in most countries, the final model failed to run in Albania and on the Swiss-French and Swiss Italian samples due to negative covariance matrices. Despite having a model that fits perfectly by definition, this error message is not unexpected with small sample sizes, in which simulation methods could cause more sampling errors (23). Moreover, the bifactor model is relatively complex and contains many parameters, which could be even more complicated with small samples. Accordingly, this could not have been prevented in countries where the total number of psychiatrists is low or the investigator had difficulties enrolling a larger sample. In Albania, the enrollment rate was 71% (n = 59 were enrolled from the total of n = 83 psychiatrists), which is considered a very high completion rate - especially among psychiatrists (24). In Switzerland, quite a large number of practitioners work in psychiatry (n = 3,848); however, we had no information about the exact number of French or Italian-speaking psychiatrists. Participants could choose the language they were most comfortable with in countries with multiple official languages. Both the Swiss French and Swiss Italian samples were quite small (n = 75 and n = 13, respectively), which exposed them to this type of error. The small sample size was probably the reason behind the Montenegrin and Azeri lower fit indices as well. Consequently, in these countries, thorough exploratory factor analysis is recommended to understand the psychometric properties of the scale. Its investigation on larger and more diverse samples (for example, with the involvement of different mental healthcare workers: e.g., psychologists, nurses, and other medical staff) could also be beneficial, particularly in those countries where the model-fit was poor.

We could not identify specific items with serious cross-loadings that were problematic in most samples. Such as item 1(‘I am more comfortable helping a person who has a physical illness than I am helping a person who has a mental illness.’) found to be weak in Singaporean medical and nursing students (10) or item 8 (‘If I had a mental illness, I would tell my friends.’) and 13 (‘Health care providers do not need to be advocates for people with mental illness.’) loaded on different factors than expected in German and Swiss healthcare workers on psychiatric wards (8). Item 11 (‘More than half of people with mental illness do not try hard enough to get better.’) was removed from the scale in a Hungarian study on psychiatrists due to poor loading across all three factors (7). In this study, different items were found to be weak in countries where the fit indices were lower than the acceptable range. Therefore, item reduction and corresponding abbreviation of the scale were not an option for finding a better fitting solution for each country.

The model-based reliability results suggest using the general factor rather than the specific factors (16). Based on our results, the disclosure and help-seeking factor were the most reliable among the specific factors, whereas this and the general factor explained a substantial proportion of the variance. These findings indicate that using the total score to quantify the overall stigmatizing attitudes and the disclosure and help-seeking subscale to gain information about the help-seeking behavior may be the best choices when we seek reliable stigma measurements, and we would like to make a comparison between countries.

The correlations among the specific factors were weak but statistically significant, supporting the notion of a shared conceptual theme. To note, the correlation coefficients in this study were lower than those in Canada (5).

In summary, this is the first study to examine the factor structure of the OMS-HC in a large sample of psychiatrists across 32 European countries by using the bifactor ESEM approach. We believe the reported findings are significant from a theoretical and an applied point of view. From a theoretical perspective, a bifactor model enables the investigation of the degree to which specific factors are independent of the general factor; thus, it may associate differently with other mental health determinants and outcomes. In addition, our model provides an upgraded version of Modgill’s three-factor model; at the same time, the bifactor approach also supports the validity of their model as comprised of three related components (5). Regarding the applied perspective, it is important to gain a better understanding of whether the OMS-HC could be used as an appropriate and reliable attitude measurement for psychiatrists. Moreover, the analyses will also help broaden the understanding of cross-cultural differences because it is important to recommend conducting psychometric analyses before evaluating the possibility of generalizing findings in different cultural contexts. However, we recommend doing this with caution, and future research must investigate the psychometric properties of the scale in countries where the model failed to run or the fit indices were not within the acceptable ranges. Finally, having a valid and reliable measure for provider stigma toward people with mental health problems in several European countries will allow us to measure and tailor anti-stigma programs for mental healthcare workers. Also, the findings may help guide future studies to evaluate and put into context the factor structure of the OMS-HC in other countries.

The findings in the study must be viewed with some limitations in mind. Firstly, we aimed to approach all practicing psychiatrists in each country; therefore, national psychiatric associations were also asked to disseminate the survey. However, despite all efforts, it was not possible to obtain nationally representative samples. Therefore, the convenience sampling approach is a possible limitation of our study. Secondly, in our research, young colleagues were overrepresented (44% of the subjects were 24–35 years of age); they were probably easier to approach with online questionnaires. Thirdly, this study examined a sample consisting solely of psychiatrists. It is uncertain if the findings apply to other healthcare workers, not only for mental healthcare providers. Lastly, most of the data were collected during the COVID-19 period. We have no data on how their attitudes would be without the challenges brought on by the pandemic. It is not yet known whether the pandemic has or not a significant effect on their stigmatizing attitudes; however, the overwhelming workload and the related stress might bring them more challenges than before (11).

5. Conclusion

Our results agree with the three-dimensional structure of the OMS-HC proposed by the scale authors after the modification and abbreviation of the scale to 15 items. In addition, however, we upgraded the investigation of the scale from a first-order factor solution to a bifactor ESEM model, which seemed more appropriate and provided us with better fit indices in a large sample of psychiatrists from several European countries. The model fit was good or acceptable in most nations; however, based on the model-based reliability results, the general factor and the disclosure and help-seeking specific factor explained substantial variance. Consequently, we recommend using the total score of the OMS-HC and probably the disclosure and help-seeking subscale scores rather than other subscales when we would like to measure the stigma in the participating countries.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The study that involved human participants was conducted in accordance with the principles of the Declaration of Helsinki. Participants provided electronic informed consent prior to completing the online survey (by clicking `yes’). The Hungarian core study was approved by the Regional and Institutional Committee of Science and Research Ethics of the Semmelweis University, Budapest, Hungary (SE-RKEB: 189/2019), followed by site approvals in participating countries. The ethics revision process for online survey projects was required in the following countries based on the written statement of the local investigators, and the study was approved by the following ethics committees. Albania: Albanian Medical Ethics Committee (Nr. 303/13). Austria: Ethics Committee of the Medical University of Graz (32-619 ex 19/20). Belarus: Ethical Committee of the Belarusian Psychiatric Association (1/2020), Belgium: Ethics committee of the University Hospital Brussels (2021/011), Croatia: Ethics Committee of the University Hospital Center Zagreb (8.1.-21/120-2, 02/21 JG), Cyprus: Cyprus National Bioethics Committee (2020.01.172.), Czech Republic: The Ethical Committee of the Third Medical Faculty, Charles University (11/2020), Estonia: Tartu University Ethics Committee, Tartu, Estonia (322/T-9), Germany: Ethikkommission der Technischen Universität München (679/20S), Greece: University of Ioannina (2638/16-7-2020), Ireland: Royal College of Physicians of Ireland Research Ethics Committee (RCPI RECSAF 134), Malta: Health Ethics Committee (HEC13/2020) Netherlands: Medical Ethics Review Committee of Erasmus Medical Center (MEC-2021-0151), Portugal: Ethical Committee of the Faculty of Medicine, University of Coimbra (CE-136/2020), Serbia: Ethics Committee of the Institute of Mental Health, Belgrade (1060/2094/1), Turkey: Research Ethics Committee of the Ministry of Health Ankara City Hospital (E1/928/2020), and United Kingdom: Pennine Care NHS Foundation.

Author contributions

DŐ was the principal investigator of the study and the major contributor to the study design, writing of the manuscript, and data analysis. PS and TM as members of the Hungarian core team, conceived and planned the study and helped in writing the manuscript. SRó being a professional statistician, verified the analytical methods, suggested and developed codes for the BESEM, helped to interpret the results, improved the manuscript with his expert comments, and also supervised the project with ZG. LB, OK, SMö, MW, MA, SB, KB, CC, EC, ED, GG, SG, II, KG, SK, KK, IK, JM, SMa, AM, NN, AP, ATP, EP-C, SRa, PR, JS-V, RS, FS, HK, AH, GA, MV, HY, and NG were essential contributors, planned the local data collection in each of their countries, translated the questionnaire to their local language(s), provided active assistance in the data collection and contributed to the manuscript writing. All authors have read and approved the final version of the manuscript and given their consent for publication.

Funding

DŐ was awarded the National Youth Talent Award in 2020 and 2021 (Ministry of Human Resources, Hungary, (NTP-NFTÖ-20-B-0134 and NTP-NFTÖ-21-B-0280) and the Prominence Award of the Kerpel-Fronius Talent Support Program of Semmelweis University in 2021 (EFOP-3.6.3-VEKOP-16-2017-00009), which covered the expenses of the 1-year account of the online survey platform and the dissemination of the study results at international congresses. SRó was supported by the Károli Gáspár University of the Reformed Church (Grant No. 20754B800).

Acknowledgments

We are sincerely thankful to Roshelle Ramkisson (consultant psychiatrist at Pennine Care NHS Foundation Trust, United Kingdom) and Cleo L. Crunelle (professor at Vrije Universiteit Brussel, Department of Psychiatry, Belgium) who provided supervision for the project in the United Kingdom and in Belgium. We express our gratitude to Anne Destrebecq (professor at Department of Biomedical Sciences for Health, University of Milan, Italy) and to Paolo Ferrara (tutor and lecturer at San Paolo Bachelor School of Nursing, San Paolo Teaching Hospital, University of Milan, Italy) for letting us use their Italian translation of the OMS-HC. Also, thanking to Anna Sofie Hansen (child psychiatrist at Aalborg University Hospital, Research Unit for Child and Adolescent Psychiatry, Aalborg, Denmark) for the preparation of the Danish data collection and Danish translation of the questionnaire. We are also grateful to all national psychiatric Associations who participated in the dissemination of the survey and special thanks to Michal Goetz (associate professor at the Department of Paediatric Psychiatry, Charles University, Prague, Czech Republic), who significantly helped in the dissemination of the survey among child psychiatrists in the Czech Republic. Finally, we would like to express our sincere gratitude to each psychiatrist colleague who participated in our study, which would not have been possible without their valuable contribution.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2023.1168929/full#supplementary-material

Abbreviations

CFI, comparative fit index; df, degree of freedom; ECV, the explained common variance; EFA, exploratory factor analysis; ESEM, exploratory structural equation modeling; KMO, Kaiser-Meyer-Olkin; OMS-HC, Opening Minds Stigma Scale for Health Care Providers; PCA, Principal component analysis; PUC, percent of uncontaminated correlations; RMSEA, root mean square error of approximation; SEM, structural equation modeling; TLI, Tucker-Lewis Index; WLSMV, diagonally weighted least squares; WRMR, Weighted Root Mean Square Residual; χ², chi-square; ωH, coefficient omega hierarchical.

References

1. Henderson, C, Noblett, J, Parke, H, Clement, S, Caffrey, A, Gale-Grant, O, et al. Mental health-related stigma in health care and mental health-care settings. Lancet Psychiatry. (2014) 1:467–82. doi: 10.1016/S2215-0366(14)00023-6

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Perry, A, Lawrence, V, and Henderson, C. Stigmatisation of those with mental health conditions in the acute general hospital setting. A qualitative framework synthesis. Soc Sci Med. (2020) 255:112974. doi: 10.1016/j.socscimed.2020.112974

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Wang, K, Link, BG, Corrigan, PW, Davidson, L, and Flanagan, E. Perceived provider stigma as a predictor of mental health service users’ internalized stigma and disempowerment. Psychiatry Res. (2018) 259:526–31. doi: 10.1016/j.psychres.2017.11.036

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Kassam, A, Papish, A, Modgill, G, and Patten, S. The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: the opening minds scale for health care providers (OMS-HC). BMC Psychiatry. (2012) 12:62. doi: 10.1186/1471-244X-12-62

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Modgill, G, Patten, SB, Knaak, S, Kassam, A, and Szeto, AC. Opening minds stigma scale for health care providers (OMS-HC): examination of psychometric properties and responsiveness. BMC Psychiatry. (2014) 14:120. doi: 10.1186/1471-244X-14-120

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Destrebecq, A, Ferrara, P, Frattini, L, Pittella, F, Rossano, G, Striano, G, et al. The Italian version of the opening minds stigma scale for healthcare providers: Validation and study on a sample of bachelor students. Community Ment Health J. (2018) 54:66–72. doi: 10.1007/s10597-017-0149-0

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Őri, D, Rózsa, S, Szocsics, P, Simon, L, Purebl, G, and Győrffy, Z. Factor structure of the opening minds stigma scale for health care providers and psychometric properties of its Hungarian version. BMC Psychiatry. (2020) 20:1–9. doi: 10.1186/s12888-020-02902-8

CrossRef Full Text | Google Scholar

8. Zuaboni, G, Elmer, T, Rabenschlag, F, Heumann, K, Jaeger, S, Kozel, B, et al. Psychometric evaluation of the German version of the opening minds stigma scale for health care providers (OMS-HC). BMC Psychol. (2021) 9:1–7. doi: 10.1186/s40359-021-00592-9

CrossRef Full Text | Google Scholar

9. Sapag, JC, Klabunde, R, Villarroel, L, Velasco, PR, Álvarez, C, Parra, C, et al. Validation of the opening minds scale and patterns of stigma in Chilean primary health care. PLoS One. (2019) 14:e0221825. doi: 10.1371/journal.pone.0221825

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Chang, S, Ong, HL, Seow, E, Chua, BY, Abdin, E, Samari, E, et al. Stigma towards mental illness among medical and nursing students in Singapore: a cross-sectional study. BMJ Open. (2017) 7:e018099. doi: 10.1136/bmjopen-2017-018099

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Őri, D, Molnár, T, and Szocsics, P. Mental health-related stigma among psychiatrists in light of COVID-19. Asian J Psychiatr. (2021) 58:102620. doi: 10.1016/j.ajp.2021.102620

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Reavley, NJ, Mackinnon, AJ, Morgan, AJ, and Jorm, AF. Stigmatising attitudes towards people with mental disorders: a comparison of Australian health professionals with the general community. Aust N Z J Psychiatry. (2014) 48:433–41. doi: 10.1177/0004867413500351

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Loch, AA, Guarniero, FB, Lawson, FL, Hengartner, MP, Rössler, W, Gattaz, WF, et al. Stigma toward schizophrenia: do all psychiatrists behave the same? Latent profile analysis of a national sample of psychiatrists in Brazil. BMC Psychiatry. (2013) 13:92. doi: 10.1186/1471-244X-13-92

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Stefanovics, E, He, H, Ofori-Atta, A, Cavalcanti, MT, Neto, HR, Makanjuola, V, et al. Cross-national analysis of beliefs and attitude toward mental illness among medical professionals from five countries. Psychiatry Q. (2016) 87:63–73. doi: 10.1007/s11126-015-9363-5

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Morin, AJ, Marsh, HW, and Nagengast, B. Chapter 10. Exploratory structural equation modeling In: GR Hancock and RO Mueller, editors. Structural Equation Modeling: A Second Course. 2nd ed. Charlotte, NC: Information Age Publishing, Inc (2013)

Google Scholar

16. Reise, SP, Scheines, R, Widaman, KF, and Haviland, MG. Multidimensionality and structural coefficient bias in structural equation modeling. Educ Psychol Meas. (2013) 73:5–26. doi: 10.1177/0013164412449831

CrossRef Full Text | Google Scholar

17. Morin, AJ, Myers, ND, and Lee, S. Modern factor analytic techniques In: G Tenenbaum and RC Eklund, editors. Handbook of sport psychology. New York, NY: John Wiley & Sons (2020). 1044–73.

Google Scholar

18. Horn, JL. A rationale and test for the number of factors in factor analysis. Psychometrika. (1965) 30:179–85. doi: 10.1007/BF02289447

CrossRef Full Text | Google Scholar

19. Awang, Z. Research methodology and data analysis. 2nd ed. Shah Alam, Selangor: UiTM Press (2012).

Google Scholar

20. McDonald, RP. Test theory: A unified treatment. New York: Psychology Press (2013).

Google Scholar

21. Corp, I. IBM SPSS Statistics for Windows. Version 26.0 ed. Armonk, NY: IBM Corp (2019).

Google Scholar

22. Muthen, LK, Muthen, B, and Muthén, M. Mplus Version 8 User's Guide. Los Angeles, CA: Muthen & Muthen (2017).

Google Scholar

23. Zitzmann, S, Walther, J-K, Hecht, M, and Nagengast, B. What is the maximum likelihood estimate when the initial solution to the optimization problem is inadmissible? The case of negatively estimated variances. Psych. (2022) 4:343–56. doi: 10.3390/psych4030029

CrossRef Full Text | Google Scholar

24. Cunningham, CT, Quan, H, Hemmelgarn, B, Noseworthy, T, Beck, CA, Dixon, E, et al. Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol. (2015) 15:1–8. doi: 10.1186/s12874-015-0016-z

CrossRef Full Text | Google Scholar