The Depression Anxiety Stress Scale-21 in Chinese Hospital Workers: Reliability, Latent Structure, and Measurement Invariance Across Genders

The Depression Anxiety Stress Scale-21 (DASS-21) is an instrument in the assessment of mental health status. The current study recruited 1,532 Chinese hospital workers [74.4% female; mean age = 31.97 (SD = 9.70) years] to examine the reliability, latent structure, and measurement invariance of the DASS-21 between genders. The Cronbach’s α values were greater than 0.90 for total score. This study examined four possible models of the DASS-21 using the confirmatory factor analysis (CFA) in Chinese hospital workers. The results from CFA revealed that the latent structure of the DASS-21 in medical staffs is best represented by a one-factor model. Then we used the one-factor model to examine measurement invariance across genders by using a multiple-group categorical CFA. All values of root mean square error approximation (RMSEA) were less than 0.08, all Comparative Fix Index (CFI) and Tucker–Lewis Index values were greater than 0.90, all ΔCFI (changes in CFI) values were less than 0.010, and ΔRMSEA (the changes in RMSEA) were less than 0.015. These findings supported the gender invariance of the DASS-21 among Chinese hospital workers.

Introduction

The Depression Anxiety Stress Scale (DASS) is a widely used instrument developed by Lovibond and Lovibond (1995b) to measure anxiety, depression, and stress. This self-reported questionnaire has 42 items. Seven items with the highest loadings from each subscale of the original DASS were selected to develop the DASS-21. The DASS was originally intended to consist of only two subscales, depression and anxiety. It covered all the core symptoms of depression and anxiety and provided the biggest discrimination between the subscales. However, during scale development, the stress subscale emerged, and more items were added to the subscale. Lovibond and Lovibond (1995b) label it “Stress” or “Tension.” The original three-factor model of Lovibond and Lovibond (1995a) includes the three factors of depression, anxiety, and stress (same as the three subscales of the DASS-21). The depression subscale is characterized by hopelessness, self-deprecation, low positive affect, and devaluation of life; the anxiety subscale is related to physiological hyperstimulation and a subjective consciousness of anxious affect, and the stress subscale is a collection of items such as relaxation difficulties, tension, impatience, irritability, and restlessness. Therefore, the depression, anxiety, and stress subscales have common characteristics, including negative affect, emotional distress, and changes in physiology in the hypothalamic–pituitary–adrenal axis (Mello et al., 2007).

Prior criticism regarding the psychometric properties of the DASS-21 has centered on its latent construct (e.g., Brown et al., 1997; Gong et al., 2010; Henry and Crawford, 2005; Le et al., 2017, although some researchers believed that this scale embodies a special, independent emotional syndrome. For example, the original three-factor model raised by Lovibond and Lovibond (1995a) and the tripartite three-factor model. The tripartite three-factor model of the DASS-21 was in accord with Watson and Clark’s (1984) tripartite conceptualization. It includes three factors – physiological hyperarousal, anhedonia, and generalized negativity. The tripartite model proposed by Watson and Clark (1984) includes three parts – a depression-type factor (characterized by low positive affect), a physiological hyperarousal factor (includes some side of anxiety), and a negative affectivity factor (contains the common mood states associated with depression and anxiety). Watson and Clark (1984) argued that the potential existence of negative affectivity could explain the high correlation in the study of mood disorders. However, negative emotion is a vague concept that is generally considered to include several aspects that are difficult to distinguish. There is also evidence that the DASS-21 is mainly related to the composition of general negative emotions. For example, Brown et al. (1997) argued that scores of the DASS-21 were primarily related to the construct of general negative affect. Duffy et al. (2005) used confirmatory factor analysis (CFA) and found that the results did not support the original three-factor structure and the tripartite three-factor model in children aged 11 to 15 years. Instead, a two-factor model consisting of “physiological hyperarousal” and “general negative affect” can fully fit the data after correlating the error terms of two items. Furthermore, both Tran et al. (2013) and Camacho et al. (2016) identified a one-factor solution in non-clinical samples by using exploratory factor analysis (EFA) (N = 221 and N = 505, respectively), and Camacho et al. (2016) suggested that the items of the DASS-21 were best represented by one factor, namely, anxious/stress–depression factor. Thus, we aim to explore the latent structure of the DASS-21 in Chinese hospital workers.

Until now, the DASS has been translated into 50 languages¹, making it widely accessible to researchers across the globe. In the last two decades, several studies in Western culture have supported the psychometric characteristics and utility of the original English version of the DASS-21, including the United States, Spain, and the United Kingdom (e.g., Brown et al., 1997; Daza et al., 2002; Crawford and Henry, 2003), and recently to Eastern culture, including China and Japan¹. Taouk et al. (2001) reported the DASS-21 for Chinese speakers in Hong Kong to use the traditional Chinese character set. Zuo and Chang (2008) directly translated the DASS from the original English version into simplified Chinese characters. However, back-translation from simplified Chinese to English was not performed, and preservation of item meaning has not been confirmed [Zuo et al. (2012), personal communication, April 29, 2012]. While the Chinese version of the DASS-21 has proven useful for field work, there are currently few data to support the psychometric characteristics of this test used in China (Chan et al., 2012). In addition, other studies have shown that the effectiveness of psychological measurement instruments may differ among populations because of differences in cultural, language expression, and values (Gjersing et al., 2010). Furthermore, there is a lack of validation studies among Asian samples on cultural differences that may make significant variations (Oei et al., 2013). For example, depression, anxiety, and stress measured by DASS-21 are considered to be different in east and west culture (Mellor et al., 2015). Marsella et al. (1985) reported that in Western populations symptoms of depression are dominated by sadness and feelings of worthlessness, whereas in non-Western populations, somatic symptoms such as sleep difficulties predominate. Leong et al. (2003) suggested that there were some cross-cultural differences in the average scores on the depression and anxiety tests. For example, some studies (e.g., Baron and Matsuyama, 1988; Shek, 1991; Chan, 1995) reported that Asian adolescents and adults have higher average scores of various depression scales than the average score previously reported for their peers in North America in the original version of the same instrument. Many studies in China also have shown that the average score of patients and non-patients on various Chinese depression scales is higher than the average score of their American counterparts (e.g., Chan and Tsoi, 1984; Chan, 1991). Also, about the DASS, Norton (2007) found that when exploring the relationship between the depression subscale and negative emotions, a small race effect was observed. When examining the relationship between the depression subscale and Beck Depression Inventory scores, it was found that African–American participants were stronger than Asian, Caucasian, or Hispanic participants. In terms of the relationship between the depression subscale and the PANAS-NA (the Negative Affect dimensions of Positive Affect Negative Affect Schedule), Caucasian and Hispanic participants were significantly stronger than African–American or Asian participants. Therefore, what the DASS-21 measures may not be exactly the same in different cultures, so it is necessary to continue to study the simplified Chinese version of the DASS-21.

The DASS also has been widely used in a variety of settings since its release in 1995, including clinical and non-clinical groups (Antony et al., 1998; Crawford and Henry, 2003; Henry and Crawford, 2005), in a variety of countries (Taouk et al., 2001; Gong et al., 2010; Chan et al., 2012), and among different age groups (Szabó, 2010; Osman et al., 2014). However, there is little research on the use of the DASS-21 among medical professionals. Lee et al. (2011) studied the internal consistency of the DASS on Korean hospital workers, but they did not study other important psychometric characteristics of the DASS-21. According to previous research, medical staff in many countries, including China, are liable to feel high levels of depression, anxiety, and stress (e.g., Caplan, 1994; Fagnani Neto et al., 2004; Ahmed et al., 2009). These studies have shown that the level of psychological distress of medical professionals is higher than that of the general public (Zhou et al., 2018). In China, Xu and Zhao (2006) reported that 28% of general hospital doctors have psychological problems, and mental health was significantly worse than that in normal adults. Researches have shown that 47.2% of medical staff suffer from depression. The mental health status of medical staff directly affects the quality of medical service and patient safety. Ignoring the mental health status of medical professionals will negatively affect the quality of medical treatment (Zhou et al., 2018). Therefore, it is necessary to ascertain the psychometric properties of the DASS-21 among Chinese medical staffs, which is conducive to its application in medical staff and has an important role in evaluating the negative emotions of medical staffs.

Differences in experiences of distress and coping mechanisms have been found between genders. Many studies have shown that females are significantly more likely to suffer from anxiety and depression disorders than males (Bruce et al., 2005; Song et al., 2008). Grant and Weissman (2007) suggested that there are gender differences in interpretation, recall, and self-reporting of afflict inventory items, and women are more likely to respond more severely to self-reporting lists (Ritvo et al., 2008). Many studies have explored gender differences on the DASS-21 total score and depression, anxiety, and stress subscales, but the results have been inconsistent. For example, Szabó (2010) reported that girls (mean age = 13.62 years) experienced significantly higher levels of depression and stress than boys. Crawford and Henry (2003) reported that women scored significantly higher than did men on the anxiety scale, depression scale, and total of the three scales, but the gender difference on the stress subscale was not statistically significant in the sample of United Kingdom adults (mean age = 40.9 years). Gong et al. (2010) found that the score on the depression subscale of boys was significantly higher than that of girls in the sample of college students (mean age = 19 years). It can be found that in different samples the gender difference results of the DASS-21 total scale and each subscale are inconsistent. Currently, the DASS-21 scoring system is the same for both men and women, but cross-group measurement invariance is necessary when using the same instrument across different groups (Reise et al., 1993). If we do not evaluate measurement invariance, the comparison of means between male and female medical stuffs could be problematic. A number of studies (Wu and Huang, 2014; Doi et al., 2018) have demonstrated that other depression and anxiety measurement scales have reliable measurement invariance (e.g., Beck Depression Inventory-II, the 7-item Generalized Anxiety Disorder Scale). Some studies have examined the measurement invariance of the DASS-21 across genders (Gomez et al., 2014; Jafari et al., 2017; Lu et al., 2018). Gomez et al. (2014) tested the measurement invariance of the DASS-21 across ratings provided by men (N = 227) and women (N = 460) in an American community sample. These authors treated item-level data as continuous and used a maximum likelihood estimation. However, the DASS-21 items are more appropriately scored on a 4-point Likert scale, and the weighted least-squares method with mean and variance adjusted (WLSMV) is used to more accurately treat the data as categorical data (Lu et al., 2018). Employing a categorical CFA (CCFA), Jafari et al. (2017) found that the DASS-21 is gender-based measurement invariance among medical students, although the psychological distress of medical students is higher than the general population. In addition, using WLSMV to analyze the data of the DASS-21, Lu et al. (2018) found support for gender-based measurement invariance in a Chinese sample. However, the samples of this study are students mainly from normal universities. To date, there have been no similar studies examining the psychometric properties of the DASS or DASS-21 in hospital workers. It is necessary to investigate the psychometric properties of the Chinese version of the DASS-21 within hospital workers to ensure that the instrument can be used as the original instrument (Scholten et al., 2017).

The current study aimed to determine the reliability and to evaluate competing models of the latent structure of the DASS-21 among a sample of Chinese hospital workers. In addition, we aimed to extend previous research by using multiple-group CFA to validate the measurement invariance of the DASS-21 across genders in Chinese hospital workers.

Materials and Methods

Participants

Of the 1,575 medical personnel from three hospitals in China asked to participate in the study, 1,532 volunteered (response rate of 97%). Of the 1,532 subjects, 472 were doctors (30.6%), 812 were nurses (52.6%), 107 were medical technicians (6.9%), 137 were administrative staff (8.9%), and 12 were categorized as “other” (0.8%). Women make up the larger portion of the sample (74.4%). In the female sample, 18.7% were doctors, 67.4% were nurses, and 5.3% were medical technicians, and 8.7% were administrative and other personnel. In the male sample, 66.2% were doctors, 9.4% were nurses, 12% were medical technicians, and 12.3% were administrative and other personnel. The average age of participants was 31.97 (SD = 9.70) years (range = 19–73 years). The average number of years working in the medical field was 9.70 (SD = 10.38) years (range = 0–55 years).

Instruments

The self-report questionnaire consists of two sections and takes approximately 10 min to complete. The first section collected data on sociodemographics including gender, age, marital status, profession, years working in the medical field, and years of education. The second section measured depression, anxiety, and stress as assessed by the DASS-21 translated by Yi et al. (2012). This 21-item scale is easy to apply in both clinical and non-clinical settings and is used to measure the negative emotions of individuals in the most recent week. Each subscale contains seven items. Participants were asked to respond on how closely the item applied to them in the past week. The scale uses the Likert four-level scoring system, with 0 to 3 points representing non-conformity (0) to very consistent (3). The higher the score, the higher the level of negative emotions.

Statistical Analyses

Data were analyzed with SPSS 20.0 (SPSS, 2011) and Mplus 7.0 (Muthén and Muthén, 1998-2012). The reliability of the DASS-21 was analyzed with SPSS 20.0. The Cronbach’s α coefficient (reliability) of the DASS-21 was considered adequate (α > 0.70). Independent-samples t tests were used to test gender difference of the total scale and three subscales. We used CFA to evaluate competing models of the latent structure of the DASS-21 by Mplus 7.0, and then the best-fit model was used to evaluate the measurement invariance. Given the response scale of the DASS-21, indicator variables were treated as categorical rather than continuous, and therefore the WLSMV estimator and theta parameterization were used in all CFAs. According to Hu and Bentler (1999), Comparative Fix Index (CFI) and the Tucker–Lewis Index value of 0.90 indicate adequate fit (and >0.95 indicates excellent fit). A root mean square error approximation (RMSEA) value of less than 0.08 indicates good fit. We used the fit indices of each model and the ΔCFI (the changes in CFI) between each model and the other three competing models to compare models. If the difference in the ΔCFI between two competing models is less than —0.01—, then we should not reject the hypothesis that no difference in fit between the two models and the more parsimonious model should be retained (Cheung and Rensvold, 2002). The four competition factor models include the original three-factor model, the tripartite three-factor model, the two-factor model (Duffy et al., 2005), and the one-factor model. The original three-factor structure includes three factors of depression, anxiety, and stress, and the tripartite three-factor model contains factors of physiological hyperarousal, anhedonia, and generalized negativity. Two-factor models concludes a general negative affect factor and a physiological hyperarousal factor, and one-factor model that all items are loading on a general factor.

To examine the measurement invariance across genders of the DASS-21, we used a multiple-group CFA in Mplus 7.0. For each comparison, three models were evaluated: M1. Configural invariance: same underlying factor structure across groups, established before conducting measurement invariance test; M2. Weak or metric invariance: invariant factor loadings across groups for gender invariance while thresholds were free to vary; M3. Strong or scalar invariance: factor loadings and thresholds are equal across groups (Dimitrov, 2010). First, we investigated whether the hypothesized one-factor model fits the data for each gender group as well as the whole sample. Second, we figured out a configural invariance model (M1). Then we used the nested model 2 (M2) to test the metric (weak) invariance, where factor loadings were equal between different genders. Finally, we tested the scalar (strong) invariance with nested model 3 (M3), where the loadings and thresholds were limited to be equal between the male and female groups (Van de Schoot et al., 2012). When M2 and/or M3 match the data as well as M1, the metric and/or scalar invariance was indicated. The establishment of both forms of invariance indicated the DASS-21 has meaningful comparability across genders (Millsap, 1998). We used the DIFTEST option in Mplus for model comparisons. It is well known that the χ² test is too sensitive to the measurement-invariance evaluation in large samples (N > 300) (Chen, 2007), and therefore, we used the ΔCFI and ΔRMSEA between the comparison and nested models to evaluate measurement invariance. The measurement invariance was indicated by ΔCFI < 0.010 and ΔRMSEA < 0.015 as described by Chen (2007) and Cheung and Rensvold (2002).

Results

Summary Statistics and Reliability of the DASS-21

Table 1 lists statistics of the DASS-21 for the whole sample, male and female groups. The Cronbach’s α coefficients of the total scale of the DASS-21 were calculated for internal consistency. The Cronbach’s α coefficient was 0.95 for the total DASS-21 scale. Independent-samples t tests revealed that men obtained significantly higher scores than did women on the depression subscale and total of the three subscales. The gender difference on the anxiety and stress subscale did not achieve statistical significance.

TABLE 1

Table 1. Summary statistics for the DASS-21 among 1,532 hospital workers in China.

Factor Structures

The WLSMV was used to perform CFA to evaluate four models of the latent structure of the DASS-21. First, we tested the original three-factor structure suggested by Lovibond and Lovibond (1995a). This model provides a satisfactory fit, but the correlation between depression factor and anxiety factor (r = 0.933), depression factor and stress factor (r = 0.951), and depression factor and stress factor (r = 0.951) suggested indistinguishability of these factors (Table 2). Next, the tripartite three-factor model developed from Watson and Clark (1984) was tested. The data fit the model well, and there were also strong interfactor correlations between the three factors (physiological hyperarousal–anhedonia = 0.894, physiological hyperarousal–generalized negativity = 0.928, anhedonia–generalized negativity = 0.980), suggesting these factors were also not distinguishable. Then, Duffy et al. (2005) two-factor models were evaluated. The fit indices for this model were receivable, and the correlation coefficient of the two factors is also high (r = 0.921). This indicates that the physiological hyperarousal and the general negative affect factors do not appear to be independent constructs. Finally, we tested the one-factor model. As Table 2 shows, all models provide an adequate fit. These results indicate that all fit indices exceeded accepted values suggested by Hu and Bentler (1999), but the interfactor correlations of the three- and two-factor models are too high, and all the ΔCFI values between competing models are smaller than —0.01—. Therefore, this study uses the one-factor model for further research. Table 3 provides the factor loadings for the one-factor model of the male and female groups. Overall, all item loadings are significant (p < 0.001) and high, ranging from 0.603 to 0.921.

TABLE 2

Table 2. Model fit indices for the four models tested.

TABLE 3

Table 3. Stand loading of each item of the DASS-21 across genders.

Measurement Invariance Between Genders

Subsequently, we tested measurement invariance of the one-factor model across the male and female groups. First, single-group CFAs were performed to check the structural validity of the DASS-21 in each gender group (see Table 4, Model C in the male/female groups). Within each gender, the one-factor model demonstrated acceptable fit. The configural invariance test was performed by studying a baseline model without constrained parameter across the two genders (M1). This model fit the data well. Thus, we tested the metric invariance model. The metric invariance model (M2) was acceptable in all fit indices. Moreover, the ΔCFI and ΔRMSEA (M1 vs. M2) were less than 0.010, supporting metric invariance and suggesting that factor loadings were invariant across genders. Given these results, we continued to test scalar invariance. This model (M3) showed receivable fit. Moreover, the ΔCFI and ΔRMSEA (M2 vs. M3) values were also within recommended ranges, supporting the scalar invariance and suggesting that factor loadings and item thresholds were invariant across genders. These results indicate that item thresholds on the latent constructs were invariant across genders. In summary, the results of multiple-group CFAs supported configural invariance, metric invariance, and scalar invariance of the DASS-21, revealing that the factor structure, factor loadings, and item thresholds were fully equivalent across genders.

TABLE 4

Table 4. Results of confirmatory factor analysis and measurement invariance across genders.

Discussion

Summary Statistics and Reliability of the DASS-21

One of the purposes of the present study was to estimate psychometric properties of the simplified Chinese version of the DASS-21 among a sample of Chinese hospital workers. Anastasi and Urbina (1997) suggested that good reliability is indicated by Cronbach’s α > 0.85. The reliability of the DASS-21 was high than this criterion (0.95), indicating that this scale is a reliable psychometric instrument with good internal consistency. Previous studies have found similar results in non-clinical and clinical groups (Antony et al., 1998), in diabetic patients (Norton, 2007), in undergraduate and graduate students (Imam, 2008), and in a young adolescent sample (Szabó, 2010). The result of the independent-samples t tests revealed that on the depression subscale and the total scale men scored significantly higher than did women. Most previous studies have suggested that women suffer more negative emotions (such as depression and anxiety) than do men (e.g., Hope and Henderson, 2014), but there are also a few studies that show no significant differences between men and women (e.g., Al Sunni et al., 2014). This article found that the levels of depression and negative emotions in the male group were significantly higher than those in the female group. The occupational ratio of male and female samples collected in this study may have resulted from the inconsistent consequences. In this study, nurses accounted for the majority of female samples (67%), whereas doctors accounted for the majority of the male groups (66.2%). While doctors usually work on the clinical frontline and may face more risks and conflicts, as a result, the level of depression of doctors may be more than that of nurses (Wu et al., 2010; Pang et al., 2014).

Factor Structures of the DASS-21

The original three-factor structure of the DASS-21 has been confirmed among diverse cultural and ethnic samples (e.g., Norton, 2007). However, there also are discrepant findings about its factor structure, ranging from one-factor (Patrick et al., 2010), two-factor (Duffy et al., 2005), and three-factor (Tully et al., 2009; Szabó, 2010) to four-factor (Szabó, 2010) structures. Hence, one purpose of this study was to explore the latent structure of the DASS-21 in Chinese hospital workers. Our study found that the three factors of the DASS-21 are conceptualized considered to be either depression, anxiety and stress; or physiological hyperarousal, anhedonia, and general negative effects; or physiological hyperarousal and general negative affect; the interfactor correlations were so large that it is impossible to establish conceptual independence. Similar results were found in the following studies. Patrick et al. (2010) evaluated the underlying structure of the DASS-21 in Australian children and adolescents sample, and the results showed that the test was difficult to distinguish between depression and anxiety and stress in this sample; it seems to measure a general distress dimension. Similarly in college students, Camacho et al. (2016) identified a one-dimension factor (anxious/stress–depression) by using exploratory factor analysis (EFA). Tran et al. (2013) found that all items of the DASS-21 were loaded on one factor in the Vietnamese female sample, and they concluded that there were no subscales or combinations that could differentiate between depression and anxiety. Similar findings were reported from a study in clinical population in Egypt. Ali and Green (2019) found that a one-factor solution was most appropriate for the DASS-21 data among Egyptian drug users.

These results suggest that it is more reasonable to view the DASS-21 as measuring only a one factor of negative affectivity or general psychological distress in Chinese hospital workers. In addition, the interfactor correlations of the models reflect limited discriminant power. Hence, perhaps the DASS-21 should be used in a sample of Chinese medical staffs with caution, and it may be best to see it as a measure of undifferentiated negative affect until further certification for the discriminate validity of depression, anxiety, and stress subscales becomes available.

Measurement Invariance Between Genders

The results of the current study sustain the measurement invariance of the DASS-21 between genders among Chinese hospital workers. The results of the single-group CFA for each gender group as well as the whole sample and configural invariance in favor of the one-factor latent structure of the DASS-21 for both males and females indicate that the DASS-21 gauged the same structures among male and female Chinese hospital workers. In addition, acceptable metric invariance indicates that the DASS-21 measures negative affective emotion in a similar manner among men and women. Furthermore, the current study found that the results of scalar invariance between genders indicate that the thresholds of each items of the DASS-21 were equal. These results suggest that comparing the scores of the DASS-21 across genders in the current sample of Chinese hospital workers is meaningful.

Because the DASS-21 scale measures negative affective emotion similarly for both females and males, it is not necessary to use different normative scores between genders. These results are consistent with the results of Gomez et al. (2014), who used an Australian sample (mean age = 47.37 years) to prove that the DASS-21 is gender-invariant, as demonstrated by Lu et al. (2018), who used a sample of Chinese college students (mean age = 19.7 years) to prove that that the DASS-21 was gender-invariant. Although the factor model used are different, Gomez et al. (2014) and Lu et al. (2018) both used the original three-factor model, and we used a one-factor model, but our current research on a sample of Chinese hospital workers reached the same conclusion, showing that application of the DASS-21 is not influenced by gender. Our findings therefore provide extra support for the DASS-21, extending prior studies in applications and research environments.

While the current study confirms the validity and reliability of the DASS-21 among genders, the study should still consider some limitations. First, half of the sample in the current study was nurses; as such, the proportion of females was significantly larger than males. While the number of males in the sample was sufficient, it should be noted that the results may not be equally representative of both genders. The second limitation of this study was the self-assessment methodology of the questionnaire, lacking clinical assessment to confirm levels of depression, anxiety, and stress. Further researches are needed to confirm latent structure and discriminate validity, sensitivity, and specificity of the DASS-21 as an effective screening instrument in various populations. Third, the sample was not recruited randomly, and thus, the interpretations of the findings are limited. Fourth, the test–retest reliability of the DASS-21 was not investigated in this study. However, as the instrument is intended to measure a state rather than trait, it would be expected that test–retest stability would be low. It would be helpful to understand the degree of stability observed when symptoms do not fluctuate.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

This study was approved by The Second Xiangya Hospital of Central South University Ethics Committee. Informed consent was obtained from all subjects.

Author Contributions

D-XW and TX conceived and designed the study. All authors were involved in the study conduction. LJ and D-XW performed the analysis and prepared the manuscript. All co-authors contributed substantially to its revision and approved the final manuscript.

Funding

This study was supported by the National Key R&D Program of China (2017YFC0820104) and Natural Science Foundation of Hunan Province (grant numbers: 2016JJ4101, 2017JJ3435, and 2017JJ4065).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We are grateful to the medical staff that participated in this study and to the professionals who helped us contact with the participants.

Footnotes

1. ^ www2.psy.unsw.edu.au/dass/

References

Ahmed, I., Banu, H., Al-Fageer, R., and Al-Suwaidi, R. (2009). Cognitive emotions: depression and anxiety in medical students and staff. J. Crit. Care 24, e1–e7. doi: 10.1016/j.jcrc.2009.06.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Al Sunni A., Latif, R., and Others. (2014). Perceived stress among medical students in preclinical years: a Saudi Arabian perspective. Saudi J. Health Sci. 3, 155–159.

Google Scholar

Ali, A. M., and Green, J. (2019). Factor structure of the depression anxiety stress Scale-21 (DASS-21): Unidimensionality of the Arabic version among Egyptian drug users. Substance Abuse Treatment Prevent. Policy 14:40. doi: 10.1186/s13011-019-0226-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Anastasi, A., and Urbina, S. (1997). Psychological Testing. Upper Saddle River, NJ: Prentice Hall/Pearson Education.

Google Scholar

Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., and Swinson, R. P. (1998). Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychol. Assess. 10, 176–181.

Google Scholar

Baron, M. R., and Matsuyama, Y. (1988). Symptoms of depression and psychological distress in United States and Japanese college students. J. Soc. Psychol. 128, 803–816. doi: 10.1080/00224545.1988.9924558

PubMed Abstract | CrossRef Full Text | Google Scholar

Brown, T. A., Chorpita, B. F., Korotitsch, W., and Barlow, D. H. (1997). Psychometric properties of the Depression Anxiety Stress Scales (DASS) in clinical samples. Behav. Res. Ther. 35, 79–89. doi: 10.1016/s0005-7967(96)00068-x

CrossRef Full Text | Google Scholar

Bruce, S. E., Yonkers, K. A., Otto, M. W., Eisen, J. L., Weisberg, R. B., Pagano, M., et al. (2005). Influence of psychiatric comorbidity on recovery and recurrence in generalized anxiety disorder, social phobia, and panic disorder: a 12-year prospective study. Am. J. Psychiatr. 162, 1179–1187. doi: 10.1176/appi.ajp.162.6.1179

PubMed Abstract | CrossRef Full Text | Google Scholar

Camacho, A., Cordero, E. D., and Perkins, T. (2016). Psychometric properties of the DASS-21 among Latina/o college students by the US-Mexico border. J. Immigr. Minor. Health 18, 1017–1023. doi: 10.1007/s10903-016-0415-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Caplan, R. P. (1994). Stress, anxiety, and depression in hospital consultants, general practitioners, and senior halth service managers. BMJ 309, 1261–1263. doi: 10.1136/bmj.309.6964.1261

PubMed Abstract | CrossRef Full Text | Google Scholar

Chan, C., and Tsoi, M. M. (1984). The BDI and stimulus determinants of cognitive-related depression among Chinese college students. Cogn. Ther. Res. 8, 501–507. doi: 10.1007/bf01173287

CrossRef Full Text | Google Scholar

Chan, D. W. (1991). The Beck depression inventory: what difference does the Chinese version make? Psychol. Assess. 3:616. doi: 10.1037/1040-3590.3.4.616

CrossRef Full Text | Google Scholar

Chan, D. W. (1995). Depressive symptoms and coping strategies among Chinese adolescents in Hong Kong. J. Youth Adolesc. 24, 267–279. doi: 10.1007/bf01537596

CrossRef Full Text | Google Scholar

Chan, R. C. K., Xu, T., Huang, J., Wang, Y., Zhao, Q., Shum, D. H. K., et al. (2012). Extending the utility of the depression anxiety stress scale by examining its psychometric properties in Chinese settings. Psychiatry Res. 200, 879–883. doi: 10.1016/j.psychres.2012.06.041

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Struc. Equ. Model. 14, 464–504. doi: 10.1080/10705510701301834

CrossRef Full Text | Google Scholar

Cheung, G. W., and Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Struc. Equ. Model. 9, 233–255. doi: 10.1207/s15328007sem0902_5

CrossRef Full Text | Google Scholar

Crawford, J. R., and Henry, J. D. (2003). The Depression Anxiety Stress Scales (DASS): normative data and latent structure in a large non-clinical sample. Br. J. Clin. Psychol. 42, 111–131. doi: 10.1348/014466503321903544

PubMed Abstract | CrossRef Full Text | Google Scholar

Daza, P., Novy, D. M., Stanley, M. A., and Averill, P. (2002). The depression anxiety stress scale-21: Spanish translation and validation with a Hispanic sample. J. Psychopathol. Behav. Assess. 24, 195–205.

Google Scholar

Dimitrov, D. M. (2010). Testing for factorial invariance in the context of construct validation. Measure. Eval. Counsel. Dev. 43, 121–149. doi: 10.1177/0748175610373459

CrossRef Full Text | Google Scholar

Doi, S., Ito, M., Takebayashi, Y., Muramatsu, K., and Horikoshi, M. (2018). Factorial validity and invariance of the 7-item generalized anxiety disorder scale (GAD-7) among populations with and without self-reported psychiatric diagnostic status. Front. Psychol. 9:1741. doi: 10.3389/fpsyg.2018.01741

PubMed Abstract | CrossRef Full Text | Google Scholar

Duffy, C. J., Cunningham, E. G., and Moore, S. M. (2005). Brief report: the factor structure of mood states in an early adolescent sample. J. Adolesc. 28, 677–680. doi: 10.1016/j.adolescence.2005.08.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Fagnani Neto, R., Obara, C. S., Macedo, P. C. M., Citero, V. A., and Nogueira-Martins, L. A. (2004). Clinical and demographic profile of users of a mental health system for medical residents and other health professionals undergoing training at the Universidade Federal de S{\~a}o Paulo. Sao Paulo Med. J. 122, 152–157. doi: 10.1590/s1516-31802004000400004

PubMed Abstract | CrossRef Full Text | Google Scholar

Gjersing, L., Caplehorn, J. R., and Clausen, T. (2010). Cross-cultural adaptation of research instruments: language, setting, time and statistical considerations. BMC Med. Res. Methodol. 10:13. doi: 10.1186/1471-2288-10-13

PubMed Abstract | CrossRef Full Text | Google Scholar

Gomez, R., Summers, M., Summers, A., Wolf, A., and Summers, J. (2014). Depression anxiety stress Scales-21: measurement and structural invariance across ratings of men and women. Assessment 21, 418–426. doi: 10.1177/1073191113514106

PubMed Abstract | CrossRef Full Text | Google Scholar

Gong, X., Xie, X., Xu, R., and Yuejia, L. (2010). Psychometric properties of the Chinese versions of DASS-21 in Chinese college students. Chin. J. Clin. Psychol. 18, 443–446.

Google Scholar

Grant, B. F., and Weissman, M. M. (2007). “Gender and the prevalence of psychiatric disorders,” in Age and Gender Considerations in Psychiatric Diagnosis, eds W. E. Narrow, M. B. First, P. J. Sirovatka, and D. A. Regier (Arlington, VA: American Psychiatric Association), 31–45.

Google Scholar

Henry, J. D., and Crawford, J. R. (2005). The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br. J. Clin. Psychol. 44, 227–239. doi: 10.1348/014466505x29657

PubMed Abstract | CrossRef Full Text | Google Scholar

Hope, V., and Henderson, M. (2014). Medical student depression, anxiety and distress outside N orth A merica: a systematic review. Med. Educ. 48, 963–979. doi: 10.1111/medu.12512

PubMed Abstract | CrossRef Full Text | Google Scholar

Hu, L., and Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struc. Equ. Model. 6, 1–55. doi: 10.1080/10705519909540118

CrossRef Full Text | Google Scholar

Imam, S. S. (2008). Depression anxiety stress scales (DASS): revisited. J. Behav. Sci. 3, 104–116.

Google Scholar

Jafari, P., Nozari, F., Ahrari, F., and Bagheri, Z. (2017). Measurement invariance of the Depression Anxiety Stress Scales-21 across medical student genders. Int. J. Med. Educ. 8, 116–122. doi: 10.5116/ijme.58ba.7d8b

PubMed Abstract | CrossRef Full Text | Google Scholar

Le, M. T. H., Tran, T. D., Holton, S., Nguyen, H. T., Wolfe, R., and Fisher, J. (2017). Reliability, convergent validity and factor structure of the DASS-21 in a sample of Vietnamese adolescents. PLoS One 12:e180557. doi: 10.1371/journal.pone.0180557

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, K. W., Kim, S. J., Park, J. B., and Lee, K. J. (2011). Relationship between depression anxiety stress scale (DASS) and urinary hydroxyproline and proline concentrations in hospital workers. J. Prevent. Med. Public Health 44, 9–13.

Google Scholar

Leong, F. T., Okazaki, S., and Tak, J. (2003). Assessment of depression and anxiety in East Asia. Psychol. Assess. 15, 290–305. doi: 10.1037/1040-3590.15.3.290

PubMed Abstract | CrossRef Full Text | Google Scholar

Lovibond, P. F., and Lovibond, S. H. (1995a). The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav. Res. Ther. 33, 335–343. doi: 10.1016/0005-7967(94)00075-u

CrossRef Full Text | Google Scholar

Lovibond, S. H., and Lovibond, P. F. (1995b). Manual for the Depression Anxiety Stress Scales. Sydney: Psychology Foundation.

Google Scholar

Lu, S., Hu, S., Guan, Y., Xiao, J., Cai, D., Gao, Z., et al. (2018). Measurement Invariance of the Depression Anxiety Stress Scales-21 Across Gender in a Sample of Chinese University Students. Front. Psychol. 9:2064. doi: 10.3389/fpsyg.2018.02064

PubMed Abstract | CrossRef Full Text | Google Scholar

Marsella, A. J., Sartorius, N., Jablensky, A., Fenton, F. F., Kleinman, A., and Good, B. (1985). Culture and Depression, Studies in the Anthropology and Cross-Cultural Psychiatry of Affect and Disorder. Berkeley, CA: University of California Press.

Google Scholar

Mello, A. F., Juruena, M. F., Pariante, C. M., Tyrka, A. R., Price, L. H., Carpenter, L. L., et al. (2007). Depression and stress: is there an endophenotype? Rev. Brasil. Psiquiatria 29, s13–s18.

Google Scholar

Mellor, D., Vinet, E. V., Xu, X., Hidayah, B. T., Mamat, N., Richardson, B., et al. (2015). Factorial invariance of the DASS-21 among adolescents in four countries. Eur. J. Psychol. Assess. 31, 138–142. doi: 10.1027/1015-5759/a000218

CrossRef Full Text | Google Scholar

Millsap, R. E. (1998). Group differences in regression intercepts: implications for factorial invariance. Multivar. Behav. Res. 33, 403–424. doi: 10.1207/s15327906mbr3303_5

PubMed Abstract | CrossRef Full Text | Google Scholar

Muthén, L. K., and Muthén, B. O. (1998-2012). Mplus User’s Guide (Version 7). Los Angeles, CA: Muthén & Muthén.

Google Scholar

Norton, P. J. (2007). Depression Anxiety and Stress Scales (DASS-21): psychometric analysis across four racial groups. Anxiety Stress Coping 20, 253–265. doi: 10.1080/10615800701309279

PubMed Abstract | CrossRef Full Text | Google Scholar

Oei, T. P., Sawang, S., Goh, Y. W., and Mukhtar, F. (2013). Using the depression anxiety stress scale 21 (DASS-21) across cultures. Int. J. Psychol. 48, 1018–1029. doi: 10.1080/00207594.2012.755535

PubMed Abstract | CrossRef Full Text | Google Scholar

Osman, Z. J., Mukhtar, F., Hashim, H. A., Latiff, L. A., Sidik, S. M., Awang, H., et al. (2014). Testing comparison models of DASS-12 and its reliability among adolescents in Malaysia. Comprehens. Psychiatr. 55, 1720–1725. doi: 10.1016/j.comppsych.2014.04.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Pang, W., Li-Qin, H. E., and Chen, Z. W. (2014). Investigation on the relationship between patient satisfaction and staff psychological health in Emergency Department. Hainan Med. J. 8, 1227–1229.

Google Scholar

Patrick, J., Dyck, M., and Bramston, P. (2010). Depression Anxiety Stress Scale: is it valid for children and adolescents? J. Clin. Psychol. 66, 996–1007.

Google Scholar

Reise, S. P., Widaman, K. F., and Pugh, R. H. (1993). Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. Psychol. Bull. 114, 552–566. doi: 10.1037/0033-2909.114.3.552

PubMed Abstract | CrossRef Full Text | Google Scholar

Ritvo, P., Myers, R., Del Giudice, M. L., Pazsat, L., Campbell, P. T., Howlett, R. I., et al. (2008). Factorial validity and invariance of a survey measuring psychosocial correlates of colorectal cancer screening in Ontario, Canada—a replication study. Cancer Epidemiol. Prevent. Biomark. 17, 3279–3283. doi: 10.1158/1055-9965.epi-08-0241

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholten, S., Velten, J., Bieda, A., Zhang, X. C., and Margraf, J. U. R. (2017). Testing measurement invariance of the Depression, Anxiety, and Stress Scales (DASS-21) across four countries. Psychol. Assess. 29, 1376–1390. doi: 10.1037/pas0000440

PubMed Abstract | CrossRef Full Text | Google Scholar

Shek, D. T. (1991). What does the Chinese version of the Beck Depression Inventory measure in Chinese students–general psychopathology or depression? J. Clin. Psychol. 47, 381–390. doi: 10.1002/1097-4679(199105)47:3<381::aid-jclp2270470309>3.0.co;2-d

CrossRef Full Text | Google Scholar

Song, Y., Huang, Y., Liu, D., Kwan, J. S., Zhang, F., Sham, P. C., et al. (2008). Depression in college: depressive symptoms and personality factors in Beijing and Hong Kong college freshmen. Comprehens. Psychiatr. 49, 496–502. doi: 10.1016/j.comppsych.2008.02.005

PubMed Abstract | CrossRef Full Text | Google Scholar

SPSS I. and Others. (2011). IBM SPSS statistics for Windows, version 20.0. New York, NY: IBM Corp, 440.

Google Scholar

Szabó, M. (2010). The short version of the Depression Anxiety Stress Scales (DASS-21): factor structure in a young adolescent sample. J. Adolesc. 33, 1–8. doi: 10.1016/j.adolescence.2009.05.014

PubMed Abstract | CrossRef Full Text | Google Scholar

Taouk, M., Lovibond, P. F., and Laube, R. (2001). Psychometric properties of a Chinese version of the short Depression Anxiety Stress Scales (DASS21). (Sydney: Report for new South Wales transcultural mental health centre, Cumberland Hospital).

Google Scholar

Tran, T. D., Tran, T., and Fisher, J. (2013). Validation of the depression anxiety stress scales (DASS) 21 as a screening instrument for depression and anxiety in a rural community-based cohort of northern Vietnamese women. BMC Psychiatry 13:24. doi: 10.1186/1471-244X-13-24

PubMed Abstract | CrossRef Full Text | Google Scholar

Tully, P. J., Zajac, I. T., and Venning, A. J. (2009). The structure of anxiety and depression in a normative sample of younger and older australian adolescents. J. Abnorm. Child Psychol. 37, 717–726. doi: 10.1007/s10802-009-9306-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Van de Schoot, R., Lugtig, P., and Hox, J. (2012). A checklist for testing measurement invariance. Eur. J. Dev. Psychol. 9, 486–492. doi: 10.1080/17405629.2012.686740

CrossRef Full Text | Google Scholar

Watson, D., and Clark, L. A. (1984). Negative affectivity: the disposition to experience aversive emotional states. Psychol. Bull. 96, 465–490. doi: 10.1037/0033-2909.96.3.465

CrossRef Full Text | Google Scholar

Wu, H., Wang, J. N., Yang, X. S., Wang, Y., and Wang, L. (2010). Prevalence and associated factors of depressive symptoms among hospital staff. 83, 905–911.

Google Scholar

Wu, P., and Huang, T. (2014). Gender-related invariance of the Beck Depression Inventory II for Taiwanese adolescent samples. Assessment 21, 218–226. doi: 10.1177/1073191112441243

PubMed Abstract | CrossRef Full Text | Google Scholar

Xu, X., and Zhao, T. (2006). The psychological health survey of medical staff in general hospital. J. China Clin. Psychol. 14, 324–325.

Google Scholar

Yi, W., Daxing, W., Xuejing, L., Hangui, L., Yuping, Y., Yuanxuan X., et al. (2012). Psychometric properties of the Chinese short version of Depression Anxiety and Stress scale in Chinese adults. Chin. J. Public Health. 17, 1436–1438. doi: 10.11847/zgggws2012-28-11-14

CrossRef Full Text | Google Scholar

Zhou, C., Shi, L., Gao, L., Liu, W., Chen, Z., Tong, X., et al. (2018). Determinate factors of mental health status in Chinese medical staff: a cross-sectional study. Medicine 97:e0113. doi: 10.1097/md.0000000000010113

PubMed Abstract | CrossRef Full Text | Google Scholar

Zuo, K. (2012). Extending the utility of the Depression Anxiety Stress scale by examining its psychometric properties in Chinese settings. Psychiatry Res. 200, 879–883. doi: 10.1016/j.psychres.2012.06.041

PubMed Abstract | CrossRef Full Text | Google Scholar

Zuo, K., and Chang, A. (2008). A Chinese translation of the DASS (simplified characters). Available online at: http://www2.psy.unsw.edu.au/groups/dass/Chinese/Chinese%20simplified.htm

Google Scholar