Jose A. Piqueras1*
Mariola Garcia-Olcina1Maria Rivera-Riquelme1Agustin E. Martinez-Gonzalez2
Pim Cuijpers3,4- 1Department of Health Psychology, Faculty of Social and Health Sciences, Center for Applied Psychology, Miguel Hernandez University, Elche, Spain
- 2Department of Developmental Psychology and Didactics, Faculty of Education, University of Alicante, San Vicente del Raspeig, Spain
- 3Department of Clinical Psychology, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- 4The Netherlands & EMGO Institute for Health and Care Research, Amsterdam, Netherlands
Emotional disorder symptoms are highly prevalent and a common cause of disability among children and adolescents. Screening and early detection are needed to identify those who need help and to improve treatment outcomes. Nowadays, especially with the arrival of the COVID-19 outbreak, assessment is increasingly conducted online, resulting in the need for brief online screening measures. The aim of the current study was to examine the reliability and different sources of validity evidence of a new web-based screening questionnaire for emotional disorder symptoms, the DetectaWeb-Distress Scale, which assesses mood (major depression and dysthymic disorder), anxiety (separation anxiety, generalized anxiety, social phobia, panic disorder/agoraphobia, and specific phobia), obsessive–compulsive disorder, post-traumatic stress disorder, suicidality (suicidal ideation, plans, and attempts), and global distress. A total of 1,499 participants (aged 8–18) completed the DetectaWeb-Distress Scale and specific questionnaires for emotional disorder symptoms, suicidal behaviors, and well-being through a web-based survey. Results indicated that a structural model of 10 correlated factors fits reasonably better in comparison to the remaining models; measurement invariance for age and gender; good internal consistency (McDonald's ω ranging from 0.65 to 0.94); and significant positive correlation with other measures of anxiety, depression, PTSD, or distress, and negative correlation with well-being measures, displaying support for convergent-discriminant validity. We also found that girls scored higher than boys on most of the subscales, and children had higher scores for social anxiety, specific phobia, panic disorder, and obsessive–compulsive symptoms, whereas adolescents scored higher on depressive symptoms, suicidality, and generalized anxiety, but the effect sizes were small to medium for all comparisons. The DetectaWeb-Distress Scale is a valid, innovative, and useful online tool for the screening and evaluation of preventive programs for mental health in children and adolescents.
Introduction
The last years of childhood and adolescence are key stages from the human developmental point of view, in which many physical, cognitive, and psychosocial changes take place, overlapping with the acquisition of new roles and responsibilities (Susman and Dorn, 2009; Steinberg, 2017). Therefore, an important development and maturation of the individual takes place, acquiring a wide and significant repertoire of personal skills that will be key to personal success later in adult life. All of these changes and demands may also be stressful, and individuals can feel emotionally overwhelmed, making them at risk of reduced mental health. Consequently, this stage is an extremely sensitive period for the development of mental health problems.
The World Health Organization report titled “Health for the World's Adolescents: A Second Chance in the Second Decade” suggests that anxiety and related disorders such as other emotional disorders (i.e., depression) are some of the most common mental disorders and most frequent causes of disease and disability in children and adolescents (World Health Organization, 2014a). Furthermore, suicide is the second leading cause of death in adolescents (World Health Organization, 2014b). More specifically, according to a recent meta-analytic review, global prevalence rates of these disorders in youth are 6.5% for anxiety and 2.6% for depressive disorders (Polanczyk et al., 2015), with a marked comorbidity between the two disorders (Cummings et al., 2014; Al-Asadi et al., 2015). Additionally, several studies among the general population of young people show that common mental disorders are one of the main risk factors for suicidal ideation and behavior (OR = 2.07–10.06) (Gili et al., 2019).
Concerning Spain, in example, Ezpeleta et al. (2007) found that between 30 and 60% of preadolescents and between 30 and 50% of adolescents presented some mental disorder, with anxiety and depression disorders as some of the most frequent disorders. Later, in 2019, Canals et al. found a prevalence of any anxiety disorder of 11.8%, with high rates of comorbidity with depression and other anxiety disorders, low use of professional support (33.3%), and high persistence of diagnosis in a 2-year follow-up (52.9%). More recently, the increase of psychological and behavioral changes, especially emotional symptoms, in Spanish children and adolescents during the early phase of COVID-19 quarantine has been reported (Francisco et al., 2020). Also, this presence of emotional symptoms, among others, at subclinical level raise the risk of subsequent development of mental disorder (Fonseca-Pedrero et al., 2020). However, the approach to adolescent mental health must contemplate not only a psychopathological view, but both the presence of difficulties and strengths (i.e., Piqueras et al., 2019; Rivera-Riquelme et al., 2019; Fonseca-Pedrero et al., 2020).
There are at least four reasons for the need for valid and reliable screening instruments for children and adolescent mental health problems (Stiffler and Dever, 2015; Tran et al., 2019).
First, it is necessary and beneficial for clinical practice to evaluate both emotional disorders and symptoms and related conditions in children and adolescents, when a first diagnostic approach is made and also to detect these symptoms in the general population (Ebesutani et al., 2012). Second, any approach to the evaluation of these disorders entails the problem of the mental health professional's lack of time to perform a diagnostic assessment, with the most common practice being the use of self-report tests to screen for these symptoms (Ebesutani et al., 2012). Third, self-report instruments have proven to be the first choice for the screening and detection of anxiety and depression with undoubted advantages over other techniques such as clinical interviews or observation techniques (Southam-Gerow and Chorpita, 2007).
Furthermore, assessment of emotional disorders is conducted increasingly online, mainly also employing self-report questionnaires (Kendrick and Pilling, 2012). Online assessment offers other advantages for participants and researchers, due to the fact that it reduces the load and allows for greater disclosure (Mogle, 2015). Recently, van Ballegooijen et al. (2016) summarized the psychometric properties of diverse online instruments evaluating anxiety and depression disorders. According to this review, there are online instruments for depression, anxiety, OCD, or PTSD for adolescents with good psychometric properties, reporting Cronbach's α ranging from 0.73 to 0.93 and evidence of convergent and criterion validity (Cuijpers et al., 2008; Keeley and Storch, 2009; Zlomke, 2009; Haavet et al., 2011). However, the review did not include any studies with children, and those found with adolescent samples were scarce. In recent times, especially with the arrival of the COVID-19 outbreak, different studies have noted the need to use online validated multi-informant and multi-problem approaches during and after home confinement (Espada et al., 2020).
Concerning the existing instruments, we shall attempt a brief description of the available tools for children's mental health professionals. According to Stiffler and Dever (2015), there are a considerable number of them, ranging from broadband multidimensional measures, through specific screenings for single vs. for multiple disorders, to those including only one indicator of overall distress.
The international community has several broad-spectrum screening measures that try to evaluate both negative and positive aspects of functioning, including an overall score of emotional problems [i.e., the Behavior Assessment System for Children and Adolescents (BASC; Reynolds and Kamphaus, 2004); the Achenbach System of Empirically Based Assessment (ASEBA; Achenbach and Rescorla, 2001, 2007); the Y-PSC-17 (Jellineck et al., 1988); the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997); or the Child and Adolescent Assessment System (SENA; Fernández-Pinto et al., 2015a,b)]. However, none of these broad-spectrum measures specifically assesses each type of anxiety, depression, and related disorder symptoms, such as PTSD, OCD, or suicidality.
Beyond these multi-component and broad-spectrum tools, there are also specific measures for only anxiety or depressive disorder symptoms. Therefore, different measures—such as the Multidimensional Anxiety Scale for Children (MASC; March et al., 1997), Children's Spence Anxiety Scale (SCAS; Spence, 1997), the Screen for Child Anxiety Related Emotional Disorders-Revised (SCARED-R; Muris et al., 1998), or the most recently published Youth Anxiety Measurement for the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5) (YAM-5, Muris et al., 2017), among others—provide valuable information not only about anxiety levels in general but also about the type of anxiety symptoms experienced by children and adolescents. Thus, these assessment tools allow examining the different types of anxiety disorders and potential comorbidity between them (Spence, 2018).
As regards depression self-reports for children and adolescents, a systematic review and meta-analysis of reliability, validity, and diagnostic utility (Stockings et al., 2015) showed that commonly used depression symptom rating scales, such as the Children's Depression Inventory (CDI, Kovacs, 1981), the Beck Depression Inventory (BDI, Beck et al., 1961), the Center for Epidemiologic Studies-Depression Scale (CES-D; Radloff, 1977), and the Reynolds Adolescent Depression Scale (RADS; Reynolds, 1986), are reliable measures of depressive symptoms among children and adolescents. However, they only provide overall scores of depression or components of depressive symptoms.
Beyond these specific tools, there are also comprehensive tests for the assessment of both anxiety and depression, such as the Revised Childhood Anxiety and Depression Scale (RCADS; Chorpita et al., 2000) and the short version RCADS-30 (Sandin et al., 2010). A recent meta-analysis indicates that the internal consistency of the different versions of the RCADS are equally high and equivalent to each other (Piqueras et al., 2017b). However, none of these versions allow one to assess some anxiety disorders, such as specific phobia, other related emotional disorders, such as dysthymia or PTSD, or suicidality.
Finally, different authors have developed measures to uniquely apprehend overall psychological distress, such as the well-known Kessler Psychological Distress Scale (Kessler et al., 2002) or the Social Emotional Distress Scale (SEDS; Dowdy et al., 2018), among others. According to Kessler et al. (2002), dimensional measures of global psychological distress have emerged, which are important to distinguish community cases based on severity rather than purely on diagnosis. From this wider framework in the assessment of mental disorders, some authors consider that subjective distress is determined largely by the presence of emotional or internalizing symptoms: anxiety, depression (Mewton et al., 2016; Dowdy et al., 2018), and suicidality (van Ballegooijen et al., 2016). Accordingly, internalizing disorders can be differentiated in two sets: distress or misery disorders such as generalized anxiety disorder (GAD), post-traumatic stress disorder (PTSD), major depression (MD), and dysthymic disorder (DD); and fear/anxiety disorders (such as panic and phobias) (Krueger, 1999; Watson, 2005; Clark and Watson, 2006). In fact, since this framework, anxiety, anxiety-related disorders such as OCD or PTSD, and depressive disorders can be collapsed into an overcharging class of “internalizing” or emotional disorders. From a broader point of view, according to relevant authors in this field (Krueger, 1999; Kessler et al., 2002; Watson, 2005; Clark and Watson, 2006), psychological distress can be conceptualized as the presence of symptoms of some of the emotional disorders without differentiating them. In fact, some authors have used the expression “depression-anxiety disorders spectrum,” “emotional disorders spectrum,” or “emotional disorders continuum” that would include different nosological entities such as misery and fear disorders, and highlighting that all these anxiety and depression-related disorders would share an internalizing factor (i.e., Gorman, 1996; Watson, 2005). In fact, a large consortium of researchers has more recently proposed the Hierarchical Taxonomy of Psychopathology (HiTOP) as an alternative to traditional categorical classifications (Kotov et al., 2017). This theoretical system has been articulated to address the limitations currently plaguing psychiatry, such as the DSM-5. Therefore, this model is a dimensional alternative to traditional nosologies for mental disorders. Their basic characteristics are (1) to consider mental health as a spectrum, that is, as a continuum between psychopathology and normality, or in other words, to consider psychopathology not as an entity in itself, but as a spectrum where different problems may share similar characteristics, and (2) to simplify the diagnostic classification, since there is a great comorbidity or overlap between disorders and this hierarchical proposal solves these difficulties.
Therefore, considering the above, our team created a new measure, the DetectaWeb-Distress Scale, the first web-based screening questionnaire for the assessment of some of the most common mental disorders among children and adolescents, such as specific subtypes of anxiety, and some of the more common anxiety-related emotional disorders such as depression, OCD, PSTD, suicidality, and overall psychological distress (Garcia-Olcina et al., 2014, 2017; Piqueras et al., 2017a, 2020). The main reason for the development of this new instrument was the need for a new specific screening instrument to detect the main emotional disorders, which should be brief, as adolescents undergo screening more easily if it is short, fast, and easy to read, according to Cuijpers et al. (2009). Furthermore, this new questionnaire is a step forward in terms of the assessment of some of the main emotional disorders through the Internet, having a potential usefulness in different fields such as child and adolescent psychopathology and clinical psychology through new technologies (i.e., for epidemiological and screening studies, diagnosis and treatment, treatment evaluation research, etc.).
The aim of the present study was the evaluation of the psychometric properties of the DetectaWeb-Distress Scale. We expected the following findings concerning different sources of validity evidence: (a) a factor structure of 10 factors to be evaluated; (b) measurement equivalence for gender and age; (c) gender and age differences in scores, such that girls and adolescents would score higher than boys and children in the total and partial scores; (d) good internal consistency reliability for the total scale and subscales; and (e) adequate validity in terms of positive and significant correlations between the total score and the subscale scores and with other patient-reported outcome measure for internalizing problems, and negative and significant correlations with different constructs.
Materials and Methods
Participants
In this study, we used a convenience sampling method. We selected eight centers of primary and secondary education through a random cluster sampling of the main counties (north, central, and south) of the province of Alicante (Valencian Community, Spain). In order to ensure that all geographic areas of the province were represented, one public and one semi-private school per county were randomly selected.
The initial sample consisted of 1,523 children and adolescents, of whom 24 were eliminated because either they did not attend school the day the survey was applied (n = 13) or they were over 18 years old (n = 11). The general inclusion criteria were the following: (a) ages 8 to 18 years and (b) being enrolled from 3rd grade of elementary education to 2nd grade of higher education. Exclusion criteria were as follows: (a) insufficient knowledge of Spanish language; (b) parents or guardians did not provide informed consent to the study, or children older than 12 did not give informed consent (compulsory according to Spanish law); and (c) students who did not attend class the day of the assessment [for further details concerning the sampling procedure, sample features, inclusion/exclusion criteria, and recruitment plan to assure sample representativeness, please see Piqueras, Garcia-Olcina et al. (2017)].
The final sample consisted of 1,499 children and adolescents (754 males) between 8 and 18 years old (M = 12.70, SD = 2.78). Most of the sample was born in Spain (93.6%). The Family Affluence Scale (Currie et al., 1997) indicated that 14.3% had a low socioeconomic status (SES), 44.1% had an intermediate SES, and 41.6% had a high SES. The distribution of children and adolescents by age and gender is presented in Table 1.
Table 1. Number and percentages of children and adolescents by age and gender.
Regardless of using convenience sampling, the adoption of the random cluster sampling method ensured the heterogeneity and representativeness of the sample. Thus, chi-square tests indicated that there was no interdependence between gender and age (χ2 = 12.29, p = 0.26), between gender and nationality (χ2 = 7.25, p = 0.29), or between gender and SES (χ2 = 0.70, p = 0.70).
Instruments
Sociodemographic Factors
The Family Affluence Scale (FAS; Currie et al., 1997) was used to measure SES. Scores from zero to seven represent categories of low (0–3), intermediate (4–5), or high (6–7) family wealth. The FAS has shown good criterion and construct validity in previous studies with adolescents (Boyce et al., 2006).
Internalizing Disorder Symptoms
The DetectaWeb-Distress Scale (Garcia-Olcina et al., 2014) is a web-based screening questionnaire created by our team. It consists of 30 items (3 items per subscale) that assess anxiety disorders, such as separation anxiety disorder (SAD), generalized anxiety disorder (GAD), specific phobia (SP), panic disorder/agoraphobia (Pd/Ag), and social phobia (SoPh); some of the main anxiety-related disorders, such as post-traumatic stress disorder (PTSD) and obsessive–compulsive disorder (OCD); mood disorders, such as major depression (MD) and dysthymic disorder (DD); suicidality (S—suicidal ideation, plans, and attempts); and a total score indicating global distress or emotional symptomatology. It is rated on a Likert-type response format (0–3). A pilot study with adolescents between 14 and 18 years old provided initial support for the reliability and validity to assess anxiety, depression, and suicidal thoughts and behaviors (Garcia-Olcina et al., 2014). The measure had good internal consistency for the Global Distress Scale (α = 0.87) and adequate correlations with related measures of anxiety and depression (RCADS: r = 0.40–87). Additionally, another study showed that, preliminarily, this measure is a reliable, valid, and useful tool to assess emotional disorders in a clinical sample (Garcia-Olcina et al., 2017), and a recent published work found that the DetectaWeb-Distress Scale is a useful measure from a diagnostic point of view, since it discriminates between people with anxiety, depression, and suicide disorders and those who do not suffer from them, presenting ROC values around 0.80 and good sensitivity and specificity for detecting the main emotional disorders (Piqueras et al., 2020). Specifically, sensitivity and specificity values of DetectaWeb-Distress total score for anxiety, depression, emotional (any anxiety or depression), and internalizing (any of them, including anxiety, depression, OCD, or PTSD) diagnosed disorders were 0.75/0.76, 0.81/0.72, 0.73/0.77, and 0.73/0.78, respectively. Additionally, a score of 25 (range = 0–90) for the total score was found to be the recommended cutoff score for a positive diagnosis. Concerning the specific subscales, the sensitivity and specificity estimates were as follows: 0.86/0.68 (SAD), 62/0.77 (GAD), 0.83/0.84 (SP), 75/0.63 (Pd/Ag), 0.62/0.82 (SoPh), 1.00/0.81 (OCD), 0.67/0.89 (PTSD), 0.75/0.96 (MD), 0.64/0.84 (DD), and 0.50/0.99 (S). The recommended cutoff scores for a positive diagnosis were respectively 4, 6, 4, 2, 5, 3, 4, 6, 4, and 4 (range = 0–9).
The Revised Child Anxiety and Depression Scale, 30-item version RCADS-30 (Sandin et al., 2010) is a reduced version of the RCADS (Chorpita et al., 2000; Sandin et al., 2009). It comprises 30 items and six subscales for evaluating symptoms of the following disorders: Pd, SoPh, SAD, GAD, OCD, and MD. Response options range from 0 (never) to 3 (always). The scale showed excellent psychometric properties in international studies and with Spanish samples (Piqueras et al., 2017b). In this sample, the McDonald's ω were as follows: Pd (0.78), SoPh (0.80), SAD (0.77), GAD (0.84), OCD (0.72), MD (0.76), and total score (0.92).
The Specific Phobia subscale of the Spence Children's Anxiety Scale; SCAS (Spence, 1997) consists of five items with four Likert alternatives (0 = never, 3 = always). We used the Spanish version, which had an average internal consistency reliability of 0.64 (Orgiles et al., 2016). The McDonald's ω for our sample was 0.62.
The Children's Revised Impact of Event Scale (CRIES; Yule, 1997) is a screening scale of PTSD for children over 8 years old. It consists of eight items rated on four-point Likert scale (0–3) and provides two subscales, four items assessing trauma-related intrusion and four avoidance. In this sample, the McDonald's ω were as follows: Intrusion (0.88), Avoidance (0.84), and total score (0.91).
Subjective Well-Being
The Mental Health Inventory (MHI; Veit and Ware, 1983) is a 38-item measure of psychological distress and well-being, developed for use in general populations, and responded from 0 (never) to 3 (always). Its factor structure for adults is psychological distress (anxiety, depression, and loss of behavioral emotional control) and psychological well-being, with general positive affect and emotional ties as subscales. Our own preliminary data from a study in progress indicate that its factor structure consists of two factors: distress (23 items) and well-being (15 items). The internal consistency (McDonald's ω) for both subscales in this sample was 0.93 and 87, respectively.
The Revised Mental Health Inventory-5 (MHI-5; Rivera-Riquelme et al., 2019) is a useful instrument to assess mental health bidimensionally, as well as to detect anxiety and depression symptoms in children and adolescents. The original MHI-5 is a brief, valid, and reliable international instrument for assessing mental health in adults (Berwick et al., 1991) as well as in children and adolescents (Marques et al., 2011). The revision of original MHI-5 consisted of the adaptation of the response format to four choices (0 = never, 1 = sometimes, 2 = often, and 3 = always). Higher scores indicate better mental health. The Revised MHI-5 has shown good psychometric properties similar to previous studies in different cultures and populations (Rivera-Riquelme et al., 2019). In this sample, we reported McDonald's ω of 0.73.
Suicidal-Related Behaviors
We used items 21 and 28 from the MHI to assess suicide-related behaviors: “During the past month, how often have you felt that others would be better off if you were dead?” and “During the past month, did you think about taking your own life?” These questions are answered in a Likert-type response scale from 0 (never) to 3 (always).
Procedure
The procedure we followed for validation of the DetectaWeb-Distress Scale was divided into four phases: (1) development of a web-based application for administration; (2) development of the instrument, according to steps for test development by (Muñiz and Fonseca-Pedrero, 2019); (3) application of the new questionnaire to a community sample; and (4) data analysis. The description of the complete procedure of the development of the instrument as well as of the DetectaWeb Project can be found elsewhere (Garcia-Olcina et al., 2014; Piqueras, Garcia-Olcina et al., 2017). DetectaWeb Project is a web-based early detection program of mental health rated on a continuum, which assesses psychological distress (DetectaWeb-Distress Scale) as well as psychological well-being (DetectaWeb-Well-being Scale) in children and adolescents. This web-based assessment protocol from the Bidimensional Mental Health Model (BMHM) has also been employed in two previous studies (Piqueras et al., 2019; Rivera-Riquelme et al., 2019).
Ethical Considerations
This study obtained the approval of the Ethical Committee for research projects (Órgano Evaluador de Proyectos, OEP) from the Vice-Rectory for Research and Technological Development of the Miguel Hernandez University (reference numbers DPS-JPR-001-10 and DPS.JPR.02.14). Children over 12 years old and their parents were requested to provide informed consent.
Statistical Analysis
All analyses were conducted with SPSS 24, EQS 6.3, and FACTOR 10.4. First, we examined item distribution and frequencies of the items. Previously, the analysis of outliers was carried out by graphically representing the results (box diagrams). Although outliers were detected, it was decided not to remove them from the sample for reasons of ecological validity. Concerning missing values, we did not have any of them, as it was mandatory to answer all the questions in order to finish the online survey.
Next, we tested a model (Model B) with nine correlated factors, a model found in the exploratory factor analysis (EFA) of the preliminary data with an adolescent sample (Garcia-Olcina et al., 2014). Then, other alternative models were examined: (i) Model A: all 30 items grouped into one general factor; (ii) Model B: nine correlated factors grouped into depression (MD and DD) and S, GAD, SoPh, SAD, SP, Pd/Ag, PTSD, and OCD; (iii) Model C: 10 correlated factors with three items per scale; (iv) Model D: 10 factors (Model C) grouped under one second-order factor that corresponds to the total scale; (v) Model E: a DSM5-based model with10 first-order factors grouped into 5 correlated second-order factors [depression (DD and D), S, anxiety (SAD, SoPh, SP, Pd/Ag, and GAD), OCD, and PTSD]; (vi) Model F: a DSM5-based model with 10 first-order factors (Model E) grouped into 5 second-order factors plus a general third-order factor (total scale).
We used correlation matrices and the Robust maximum likelihood (ML) method in all cases (EQS 6.3). We calculated the following indices as goodness-of-fit measures: Satorra-Bentler chi-square; S-B χ2/degrees of freedom (χ2/df) ratio (Chau, 1997); Root Mean Square Error of Approximation (RMSEA) (Schumacker and Lomax, 2004); Standardized Root Mean Square Residual (SRMR), Comparative Fit Index (CFI), Goodness of Fit Index (GFI), Adjusted Goodness of Fit Index (AGFI) (Bentler, 1990), and Akaike Information Criterion (AIC) (Jöreskog and Sörbom, 1993).
Later, we tested whether the DetectaWeb-Distress Scale exhibits metric invariance. Progressive evaluation of Factor Invariance (FI) was conducted through the Mean and Covariances Structures (MACS) method, as recommended by Byrne et al. (2009). The reported fit indices were RMSEA and CFI, which are the main indicators to evaluate FI. According to Cheung and Rensvold (2002), invariance between samples is admissible when the difference between the CFIs (Increment CFI) is ≤0.01 with respect to the previous model. The CFI is complemented by the AIC, which is interpreted as absence of FI when there is a considerable increase in this index. Mardia (1974) test was employed to assess multivariate normality of data, in which values lower than 5.00 are indicative of normality. The estimation method used was Robust ML.
When there is strong measurement invariance, the comparison of factor means across groups is permissible (Dimitrov, 2012). Consequently, we calculated age and gender differences. We also estimated Cohen (1988) d index (standardized mean difference), which allows evaluating the effect size (ES) of the obtained differences. McDonald (1999) ω was used to estimate the internal consistency of the DetectaWeb-Distress total scale and subscales; it is a better estimator of reliability than Cronbach's α (Dunn et al., 2014).
Finally, convergent-discriminant validity was evaluated by calculating the correlation coefficients between the scores on the DetectaWeb-Distress Scale and different well-established measures. Cohen's criteria were used to estimate the ES of the correlations (Cohen, 1988; Lipsey and Wilson, 2001).
Results
Item Analysis and Reliability
The frequencies of item responses indicated that all response options had been chosen. Items 7, 8, and 9 (S) obtained less frequent responses of “often” and “always.”
The mean item response ranged from 0.07 (Item 9) to 2.11 (Item 22), and standard deviations ranged from 0.34 (Item 9) to 1.08 (Item 11). The mean response of the items was 0.71 (SD = 0.37), which is noticeably lower than the average theoretical point of the scale, 1.5. With respect to the values of the correlations, the item total of corresponding subscale corrected index did not find any value <0.30 (r) (Nunnally, 1994) (see Table 2).
Table 2. Means (M), standard deviation (SD), corrected item total of corresponding subscale correlation (r), Cronbach α if item eliminated (α-i), and reliability (McDonald's ω) of the Detectaweb-Distress Scale.
Confirmatory Factor Analysis (CFA)
As can be seen in Table 3, goodness-of-fit indices indicated that the best models were Models B and C, with CFI, GFI, and AGFI equal to or >0.90, and RMSEA < 0.05. Models A and D did not receive empirical support: in Model A, RMSEA was >0.60, and in both cases, CFI was <0.85. The other models showed adequate fit indices, even the DSM5-based models (E and F), due to the fact that the goodness-of-fit indices indicated that these models fit the data acceptably. However, Model C had the best fit indices as well as corresponding the best with the theoretical model we considered (10 different but correlated emotional disorders).
Table 3. Goodness-of-fit indices of confirmatory factor analysis.
Table 2 (last column) shows the degree of relationship (standardized weights) for each item on its corresponding factor. All item weights are above or near 0.60 (0.45–0.90).
Factor Invariance
First, the FI of Model C was tested across age (see Table 4). Mardia's test indicated non-normal data. Configural, weak, and strong invariance models were tested using Robust ML. The configural invariance model fit adequately with Robust ML due to the fact that the CFI was 0.91, which is larger than 0.90 (Schermelleh-Engel et al., 2003). The weak invariance model also showed an adequate fit, because the change in the CFI was not >0.01, and the AIC increased slightly. Next, the strong invariance model was also confirmed, because the change in the CFI was <0.01, and the AIC increased only slightly. Finally, the strict FI model did not fit adequately, because the reduction of the CFI was higher than 0.01, and the AIC increased considerably.
Table 4. Fit indices of invariance models across age and gender.
Second, we tested the FI across gender (see Table 4). The configural invariance model fit adequately due to a CFI value of 0.916, which indicates adequate fit (Schermelleh-Engel et al., 2003). The weak invariance model showed the same CFI, and the AIC did not increase considerably. The analysis of strong invariance indicated adequate fit because the CFI did not change, but the AIC increased slightly. Lastly, the strict FI model was tested and showed that the CFI decreased around 0.01, and the AIC increased.
Gender and Age Differences on the DetectaWeb-Distress Scale
Table 5 shows the means, standard deviations, and scale scores based on gender and age. Overall, girls scored higher than boys, but the differences were significant only for MD, SAD, SP, Pd/Ag, GAD, and OCD. Regarding age, children scored higher than adolescents on SAD, SP, Pd/Ag, and OCD, whereas adolescents displayed higher scores on MD, DD, S, and GAD. The ESs were small for all comparisons with the exception of SP, which reached a medium magnitude, with higher scores for females (d = 0.51).
Table 5. Means and standard deviations of DetectaWeb-Distress Scale scores based on gender and age.
Estimations of Reliability
The reliability estimations were calculated with McDonald's ω. The reliability for the overall distress score was 0.91. The remaining values were between 0.65 and 0.94 (see Table 2). Moreover, as the DSM5-based factorial model showed a good fit, the calculation and use of total scores for depressive symptoms (MD + DD) and anxiety symptoms (sum of SAD, GAD, SoPh, Pd/Ag, and SP) was justified, which resulted in an internal consistency of 0.82 and 87, for depressive and anxiety symptoms, respectively.
Convergent-Discriminant Validity
We analyzed the bivariate correlations (Pearson's coefficients) between subscale scores of the DetectaWeb-Distress Scale and other patient-reported outcome measure for internalizing problems (Table 6). Significant and positive relationships between all scores were found (p < 0.01). First, correlations between the DetectaWeb-Distress subscale scores ranged from 0.43 (S) to 0.71 (SoPh), with a general trend to find a lower relationship of S with the other subscales (r = 0.06–0.43) than for other associations (0.17–0.59). Second, concerning the relationship between the RCADS-30 and the DetectaWeb-Distress Scale scores, correlation coefficients ranged between 0.35 and 0.82. Third, the relationships between the score on DetectaWeb-Distress SP subscale and the equivalent SCAS and SP subscale were positive, and the PTSD subscale of the DetectaWeb-Distress Scale and the CRIES scores were also positively correlated. Finally, the score on S of the DetectaWeb-Distress Scale positively correlated with items related to S and the Distress score of the MHI, as well as negatively with well-being of the MHI and the MHI-5 (see Table 6).
Table 6. Convergent-discriminant validity of the DetectaWeb-Distress Scale.
Discussion
The aim of this study was to examine the the reliability and different sources of validity evidence of the DetectaWeb-Distress Scale in children and adolescents.
First, an item analysis was performed, which showed that the mean scores of the items were adequate, as they were close to the midpoint of the scale. In addition, all response options for the items were chosen, with a limited range of responses for those items that corresponded to the suicidality factor. Concerning item total of corresponding subscale correlations, we did not find any value lower than 0.30.
The CFA tested the nine-factor model reported in previous studies (Garcia-Olcina et al., 2014; Piqueras et al., 2020), as well as other theory-based models. However, the best fit to data was for the 10 correlated-factor model, which included MD, DD, S, SAD, SoPh, SP, Pd/Ag, GAD, OCD, and PTSD. The explanation for this finding is that, in this study, the factor of depressive disorders was divided into two related disorders: MD and DD. Our model is equivalent to other measures reporting multidimensional models where each factor corresponds to the dimension that it aims to measure (e.g., RCADS or SCAS). However, our instrument includes more emotional disorder symptoms with a lower number of items, showing equivalent psychometric properties to previous measures. Anyway, a more detailed discussion of the model or models that appear to underlie the instrument in comparison to other possible models that have not been tested in the present study deserves some mention.
So, on the one hand, it seems that our results also support two DSM5-based models tested in the study (Models D and E) and, therefore, they seem to support the DSM-5 proposal for the existence of a suicidal behavior disorder, since suicidal symptoms form a distinct factor from other emotional symptoms, although a related factor. However, it would be possible that a model that included depressive and suicidal symptoms within the same factor would also fit the data well. This model has not been tested in the present study and should await further research. In this sense, it must be remembered that the proposal of a suicidal behavior disorder is a controversial issue, and in fact, in the DSM-5, it was included in the chapter dedicated to conditions for further study.
On the other hand, from a theoretical point of view, the psychological theoretical model that could framework our finding could be the accumulative tradition of alternative proposal focused on solving the shortcomings of traditional taxonomies in the form of a quantitative nosology, an evidence-based organization of psychopathology (e.g., Krueger, 1999; Achenbach and Rescorla, 2001, 2007; Kessler et al., 2002; Watson, 2005; Clark and Watson, 2006; Kotov et al., 2017). These quantitative nosologies, rather than being constructed from the top down, have emerged from the independent work of multiple research groups trying to understand the natural organization of psychopathology (Kotov et al., 2017). According to this HiTOP model, the dimensions or spectra that make it possible to obtain the super spectra or high-order dimensions are six (spectra). They are established to categorize the different subfactors of symptoms. The so-called internalizing spectrum includes sexual problems, eating pathology, fear, distress, and mania (subfactors). Fear subfactor includes SoPh, Pd/Ag, SP, SAD, and OCD, while Distress subfactor includes MD, DD, GAD, PTSD, and borderline personality disorder. All these syndromes or disorders emerge from symptom components and maladaptive traits (Components) and symptoms (Signs and Symptoms) (see Figure 2. Spectra of the Hierarchical Taxonomy of Psychopathology in Kotov et al., 2017).
Concerning the factorial invariance analyses with the MACS method, in general, our results showed factor structure equivalence across age and gender. Consequently, it allows us not only to compare mean scores of items and factors but also to conclude that the factor structure is equivalent in both groups (Dimitrov, 2010). In summary, strong measurement invariance was found for both gender and age variables, which indicates equal factor loadings and equal indicator intercepts (i.e., indicator means) across groups. This finding implies that, when strong measurement invariance is shown, the comparison of factor means across groups is permissible (Dimitrov, 2012).
Regarding gender and age differences, we found that girls scored higher than boys on most of the subscales, which is consistent with previous studies (Garcia-Olcina et al., 2014; Lewis et al., 2019). In terms of age, children had higher scores for SAD, SP, Pan/Ag, and OCD symptoms, whereas adolescents scored higher on MD, DD, S, and GAD. These findings are consistent with previous studies that point out that age is a conditioning factor of the different anxiety and depression symptoms (Canals et al., 2019). These results are also compatible with the HiTOP Model, which would suggest that the Fear subfactor (SoPh, Aga, SP, SAD, PD, and OCD) would have an earlier onset, while the Distress subfactor (MD, DD, GAD, and PTSD) would be more prevalent among adolescents (Kotov et al., 2017).
Internal consistency reliability for the Overall Distress Scale (0.91) was higher than the 0.70 recommended by Nunnally (1994). This reliability value is equivalent to those reported in previous studies using web-based measures of anxiety and depression with internal consistencies between 0.88 and 0.95 (van Ballegooijen et al., 2016), as well as the reliability reported for the RCADS and SCAS (mean α values of 0.93 and 0.92, respectively). The DetectaWeb-Distress subscales obtained values between 0.62 and 0.94. These values are equivalent to those reported for measures of anxiety and depression such as the RCADS, ranging from 0.74 to 0.85 (Piqueras et al., 2017b), and the SCAS, ranging from 0.64 to 0.80 (Orgiles et al., 2016).
As for evidence of validity, our results showed good convergent-discriminant validity, with significant correlations between the DetectaWeb-Distress Scale and other measures of related constructs. The correlation with the RCADS was r = 0.82, which is consistent with other results reported with measures of anxiety and depression symptomatology in youth, such as the Epidemiological Studies Depression Scale (CESD) and the Major Depression Inventory (MDI) with correlations of 0.88, or the Depression and Anxiety subscales of the Depression, Anxiety, and Stress Scale (DASS), with a correlation of 0.83 (Cuijpers et al., 2008; Zlomke, 2009). Additionally, correlations between the DetectaWeb-Distress Scale subscales and the RCADS subscales were significant, showing ESs between moderate and large (r = 0.35–0.67). These findings are consistent with those reported in previous studies. For example, Zlomke (2009) found moderate correlations between the DASS scales and the Penn State Worry Questionnaire (PSWQ), ranging from 0.28 to 0.49.
The DetectaWeb-Distress Scale subscales that had no analog dimensions in the RCADS also correlated strongly with other specific measures that we included. SP showed a high correlation with the homolog subscale of the SCAS (r = 0.63). This finding was similar to that of Orgiles et al. (2012), who obtained a moderate but significant correlation between the SCAS subscale for Physical Injury Fears and the STAI-C. Our subscale of PTSD symptoms showed a significant moderate correlation with the CRIES total score (r = 0.35), which coincides with the study by Zlomke (2009) in which there was a correlation of 0.49 between the Stress subscale of the DASS and the PSWQ. We also found a high correlation between the S subscale and the MHI items for suicide.
In summary, our first hypothesis concerning the correlated 10-factor solution was supported. The second hypothesis was confirmed: factorial invariance across gender and age was revealed. The third hypothesis was also confirmed, as we obtained age and gender differences in the symptom subscales, as expected. With regard to our fourth hypothesis, we obtained good internal consistency reliability, which is equivalent to the results of other web-based questionnaires (van Ballegooijen et al., 2016). Finally, the fifth hypothesis was confirmed due to the fact that correlations with equivalent measures that assess the same or related constructs were significant and positive.
Some methodological limitations should be noted. First, this scale seems to have reliable and valid indicators for emotional symptomatology in youth, but it is not up to date with the current 5th edition of the DSM (DSM-5, American Psychological A.ssociation, 2013). DSM-5 has made two main changes with regard to the anxiety disorders section: (1) selective mutism is now included as an anxiety disorder, and (2) OCD and PTSD have been removed from the section as they are no longer considered as pure anxiety syndromes. However, the DetectaWeb-Distress Scale would allow the calculation of different scores compatible with the current DSM-5: a general indicator of depressive disorder symptoms and specific symptoms of unipolar mood disorders (MD and DD), a total index of anxiety disorder symptoms plus each specific symptomatology (SAD, GAD, SoPh, Pd/Ag, SP), as well as OCD, PTSD, and S indices, which can be considered anxiety and depression-related disorders following DSM-5 rationale. Second, this study should be extended to clinical samples in order to provide clinical validity and to allow clearly differentiating between healthy and anxious/depressive children and adolescents, although a recent study addressed this issue (Piqueras et al., 2020). Third, according to the conceptualization of mental health as a continuum of psychological distress and well-being, future studies should provide data concerning both poles of the mental health continuum. Fourth, the DetectaWeb-Distress scores should be compared with clinical diagnostic interviews in order to examine diagnostic validity, also tested in Piqueras et al. (2020). Anyway, it should be mentioned that this measure may be useful to screen and detect, but it is not a tool to diagnose emotional disorders. Fifth, cross-cultural studies are needed to determine the psychometric properties of the scale across languages and cultures. It is expected that these findings may be generalized to non-Spanish-speaking children and adolescents (i.e., data should be replicated with other Latino groups from Latin America or USA, as well as with English-speaking participants). Finally, the present study has got some other constraints typically found with the use of self-report measures, such as the convenience of using other methods to generalize our findings (e.g., multiple informants) and the absence of social desirability scales or of infrequency scales to detect random responses, among other bias. Despite these limitations, we note that this measure has several strengths, such as its brevity and being one of the first measures developed specifically for online use.
Conclusions
In summary, support was found for the DetectaWeb-Distress Scale as a valid and useful web-based instrument for the early screening and identification of anxiety and depression and some of the more common related emotional symptoms (depression, OCD, or PTSD), as well as for suicidality, in children and adolescents. It is the first one specifically developed for use through the Internet. Furthermore, the scale has potential as a useful instrument in the implementation of preventive interventions for anxiety–depression and related symptoms, as well as for the promotion of well-being and mental health.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics Statement
The studies involving human participants were reviewed and approved by Ethical Committee for research projects (Órgano Evaluador de Proyectos, OEP) from the Vice-Rectory for Research and Technological Development of the Miguel Hernandez University (reference numbers DPS-JPR-001-10 and DPS.JPR.02.14). Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.
Author Contributions
JP conceived the study, participated in the data collection, led the preparations, analyzed the data, and wrote the first draft of the manuscript. MG-O, MR-R, and AM-G participated in the recruitment of the sample and data collection. PC participated in the study conception and in the data analysis. All authors contributed to interpreting the data, helped to draft and revise the manuscript, and read and approved the final manuscript.
Funding
This study was supported by research grants conceded to the JP by the Vicerrectorado for Research and Technological Development from the Miguel Hernández University (BANCAJA-UMH, call 2010), the Department of Health Care (Consejeria de Sanidad) from the Valencian Community (SMI 10/2014), and the Vicerrectorado for Research from the Miguel Hernández University (YS0017VF); and a grant for the hiring of a pre-doctoral research assistant awarded to the MR-R (ACIF/2015/155; VALi+d Program; Department of Education, Research, Culture and Sport from the Valencian Community).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We would like to thank the collaboration of eight primary and secondary schools from the province of Alicante (Valencian Community, Spain). We are especially grateful for the collaboration of the management teams of centers, the participants, and their parents or guardians.
References
Achenbach, T. M., and Rescorla, L. A. (2001). Manual for the ASEBA School-Age Forms and Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, and Families.
Achenbach, T. M., and Rescorla, L. A. (2007). Multicultural Supplement to the Manual for the ASEBA School-Age Forms and Profiles. Burlington, VT: University of Vermont: Center for Children, Youth, and Families.
Al-Asadi, A. M., Klein, B., and Meyer, D. (2015). Multiple comorbidities of 21 psychological disorders and relationships with psychosocial variables: a study of the online assessment and diagnostic system within a web-based population. J. Med. Internet Res. 17:e55. doi: 10.2196/jmir.4143
American Psychological A.ssociation (2013). Diagnostic and Statistical Manual of Mental Disorders, 5th Edn. Washington, DC: American Psychiatric Association. doi: 10.1176/appi.books.9780890425596
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., and Erbaugh, J. (1961). An inventory for measuring depression. Arch. Gen. Psychiatry 4, 561–571. doi: 10.1001/archpsyc.1961.01710120031004
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychol. Bull. 107, 238–246. doi: 10.1037/0033-2909.107.2.238
Berwick, D. M., Murphy, J. M., Goldman, P. A., Ware, J. E., Barsky, A. J., and Weinstein, M. C. (1991). Performance of a five-item mental health screening test. Med. Care 29, 169–176. doi: 10.1097/00005650-199102000-00008
Boyce, W., Torsheim, T., Currie, C., and Zambon, A. (2006). The family affluence scale as a measure of national wealth: validation of an adolescent self-report measure. Soc. Indic. Res. 78, 473–487. doi: 10.1007/s11205-005-1607-6
Byrne, B. M., Oakland, T., Leong, F. T. L., van de Vijver, F. J. R., Hambleton, R. K., Cheung, F. M., et al. (2009). A critical analysis of cross-cultural research and testing practices: implications for improved education and training in psychology. Train. Educ. Prof. Psychol. 3, 94–105. doi: 10.1037/a0014516
Canals, J., Voltas, N., Hernández-Martínez, C., Cosi, S., and Arija, V. (2019). Prevalence of DSM-5 anxiety disorders, comorbidity, and persistence of symptoms in Spanish early adolescents. European Child and Adolescent Psychiatry 28, 131–143. doi: 10.1007/s00787-018-1207-z
Chau, P. Y. K. (1997). Reexamining a model for evaluating information center success using a structural equation modeling approach. Dec. Sci. 28, 309–334. doi: 10.1111/j.1540-5915.1997.tb01313.x
Cheung, G. W., and Rensvold, R. B. (2002). Evaluating goodness-of- fit indexes for testing measurement invariance. Struct. Equation Model. 9, 233–255. doi: 10.1207/S15328007SEM0902_5
Chorpita, B. F., Yim, L., Moffitt, C. E., Umemoto, L. A., and Francis, S. E. (2000). Assessment of symptoms of DSM-IV anxiety and depression in children: a revised child anxiety and depression scale. Behav. Res. Ther. 38, 835–855. doi: 10.1016/S0005-7967(99)00130-8
Clark, L. A., and Watson, D. (2006). Distress and fear disorders: an alternative empirically based taxonomy of the “mood” and “anxiety” disorders. Br. J. Psychiatry 189, 481–483. doi: 10.1192/bjp.bp.106.03825
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. Statistical Power Analysis for the Behavioral Sciences.
Cuijpers, P., Boluijt, P., and van Straten, A. (2008). Screening of depression in adolescents through the Internet : sensitivity and specificity of two screening questionnaires. Eu. Child Adolesc. Psychiatry 17, 32–38. doi: 10.1007/s00787-007-0631-2
Cuijpers, P., Smits, N., Donker, T., Ten Have, M., and de Graaf, R. (2009). Screening for mood and anxiety disorders with the five-item, the three-item, and the two-item mental health inventory. Psychiatry Res. 168, 250–255. doi: 10.1016/j.psychres.2008.05.012
Cummings, C. M., Caporino, N. E., and Kendall, P. C. (2014). Comorbidity of anxiety and depression in children and adolescents: 20 years after. Psychol. Bull. 140, 816–845. doi: 10.1037/a0034733
Currie, C. E., Elton, R. A., Todd, J., and Platt, S. (1997). Indicators of socioeconomic status for adolescents: the WHO health behaviour in school-aged children survey. Health Educ. Res. 12, 385–397. doi: 10.1093/her/12.3.385
Dimitrov, D. M. (2010). Testing for factorial invariance in the context of construct validation. Meas. Eval. Counsel. Dev. 43, 121–149. doi: 10.1177/0748175610373459
Dimitrov, D. M. (2012). Statistical Methods for Validation of Assessment Scale Data in Counseling and Related Fields. Alexandria, VA: American Counseling Association, John Wiley and Sons.
Dowdy, E., Furlong, M. J., Nylund-Gibson, K., Moore, S., and Moffa, K. (2018). Initial validation of the social emotional distress survey–secondary to support complete mental health screening. Assess. Effect. Intervent. 43, 241–248. doi: 10.1177/1534508417749871
Dunn, T. J., Baguley, T., and Brunsden, V. (2014). From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. Br. J. Psychol. 105, 399–412. doi: 10.1111/bjop.12046
Ebesutani, C., Bernstein, A., Chorpita, B. F., and Weisz, J. R. (2012). A transportable assessment protocol for prescribing youth psychosocial treatments in real-world settings: reducing assessment burden via self-report scales. Psychol. Assess. 24, 141–155. doi: 10.1037/a0025176
Espada, J. P., Orgilés, M., Piqueras, J. A., and Morales, A. (2020). Las buenas prácticas en la atención psicológica infanto-juvenil ante el COVID-19. Clínica y Salud. 31, 109–113. doi: 10.5093/clysa2020a14
Ezpeleta, L., Guillamón, N., Granero, R., de la Osa, N., Domènech, J. M., and Moya, I. (2007). Prevalence of mental disorders in children and adolescents from a Spanish slum. Soc. Sci. Med. 64, 842–849. doi: 10.1016/j.socscimed.2006.10.031
Fernández-Pinto, I., Santamaría, P., Sánchez-Sánchez, F., Carrasco, M. A., and del Barrio, V. (2015a). SENA. Sistema de Evaluación de Niños y Adolescentes. Manual técnico [SENA. Assessment System for Children and Adolescents.Technical handbook]. Madrid: TEA Ediciones.
Fernández-Pinto, I., Santamaría, P., Sánchez-Sánchez, F., Carrasco, M. A., and del Barrio, V. (2015b). SENA. Sistema de Evaluación de Niños y Adolescentes. Manual de Aplicación, Corrección e Interpretación [SENA. Assessment System for Children and Adolescents. Handbook for Administration, Scoring, and Interpretation]. Madrid: TEA Ediciones.
Fonseca-Pedrero, E., Ortuño-Sierra, J., and Pérez-Albéniz, A. (2020). Emotional and behavioural difficulties and prosocial behaviour in adolescents: a latent profile analysis. Rev. Psiquiatría Salud Mental 13, 202–212. doi: 10.1016/j.rpsm.2020.01.002
Francisco, R., Pedro, M., Delvecchio, E., Espada, J. P., Morales, A., Mazzeschi, C., et al. (2020). Psychological symptoms and behavioral changes in children and adolescents during the early phase of COVID-19 quarantine in three european countries. Front. Psychiatry 11:570164. doi: 10.3389/fpsyt.2020.570164
Garcia-Olcina, M., Piqueras, J. A., and Martínez-González, A. (2014). Datos preliminares de la validación del Cuestionario de Detección vía Web para los trastornos emocionales (DETECTA-WEB) en adolescentes españoles. [Preliminary data of validation of the web-based screening questionnaire for emotional mental disorders (DETECTA-WEB) in Spanish adolescents]. Rev. Psicol. Clín. Con Niños Adolescentes 1, 69–77. Retrieved from: https://www.revistapcna.com/sites/default/files/14_08.pdf
Garcia-Olcina, M., Rivera-Riquelme, M., Canto-Díez, T., Berenguer, M., Bustamante, R., and Piqueras, J. A. (2017). Detección online de trastornos emocionales en población clínica de niños y adolescentes: escala detectaweb-malestar. [Online detection of emotional disorders in the clinical population of children and adolescents: DetectaWeb-Distress Scale]. Rev. Psicol. Clín. Con Niños Adolescentes 4, 35–45. Retrieved from: https://www.revistapcna.com/sites/default/files/17-18_1.pdf
Gili, M., Castellví, P., Vives, M., de la Torre-Luque, A., Almenara, J., Blasco, M. J., et al. (2019). Mental disorders as risk factors for suicidal behavior in young people: a meta-analysis and systematic review of longitudinal studies. J. Affect. Disord. 245, 152–162. doi: 10.1016/j.jad.2018.10.115
Goodman, R. (1997). The strengths and difficulties questionnaire: a research note. J. Child Psychol. Psychiatry 38, 581–586. doi: 10.1111/j.1469-7610.1997.tb01545.x
Gorman, J. M. (1996). Comorbid depression and anxiety spectrum disorders. Depress. Anxiety 4, 160–168. doi: 10.1002/(SICI)1520-6394(1996)4:4<160::AID-DA2>3.0.CO;2-J
Haavet, O. R., Sirpal, M. K., Haugen, W., and Christensen, K. S. (2011). Diagnosis of depressed young people in primary health care-a validation of HSCL-10. Fam. Pract. 28, 233–237. doi: 10.1093/fampra/cmq078
Jellineck, M. S., Murphy, J. M., Robinson, J., Feins, A., Lamb, S., and Fenton, T. (1988). Pediatric symptom checklist: screening school-age children for psychosocial dysfunction. J. Pediatr. 112, 201–209. doi: 10.1016/S0022-3476(88)80056-8
Jöreskog, K. G., and Sörbom, D. (1993). LISREL 8: Structural Equation Modeling With the SIMPLIS Command Language. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Keeley, M. L., and Storch, E. A. (2009). Anxiety disorders in youth. J. Pediatr. Nurs. 24, 26–40. doi: 10.1016/j.pedn.2007.08.021
Kendrick, T., and Pilling, S. (2012). Common mental health disorders - Identification and pathways to care: NICE clinical guideline. Br. J. Gen. Pract. 62, 47–49. doi: 10.3399/bjgp12X616481
Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L. T., et al. (2002). Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol. Med. 32, 959–976. doi: 10.1017/S0033291702006074
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., et al. (2017). The Hierarchical Taxonomy of Psychopathology (HiTOP): a dimensional alternative to traditional nosologies. J. Abnorm. Psychol. 126, 454–477. doi: 10.1037/abn0000258
Kovacs, M. (1981). Rating scales to assess depression in school-aged children. Acta Paedopsychiatr. 46, 305–315.
Krueger, R. F. (1999). The structure of common mental disorders. Arch. Gen. Psychiatry 56, 921–926. doi: 10.1001/archpsyc.56.10.921
Lewis, A. J., Sae-Koew, J. H., Toumbourou, J. W., and Rowland, B. (2019). Gender differences in trajectories of depressive symptoms across childhood and adolescence: a multi-group growth mixture model. J. Affect. Disord. 260, 463–472. doi: 10.1016/j.jad.2019.09.027
Lipsey, M. W., and Wilson, D. B. (2001). Practical meta-analysis. Appl. Soc. Res. Methods Series 49:264.
March, J. S., Parker, J. D., Sullivan, K., Stallings, P., and Conners, C. K. (1997). The multidimensional anxiety scale for children (MASC): factor structure, reliability, and validity. J. Am. Acad. Child Adolesc. Psychiatry 36, 554–565. doi: 10.1097/00004583-199704000-00019
Mardia, K. V. (1974). Applications of some measures of multivariate skewness and kurtosis in testing normality and robustness studies. Sankhyā 36, 115–128.
Marques, S. C., Pais-Ribeiro, J. L., and Lopez, S. J. (2011). Use of the “mental health inventory – 5” with portuguese 10-15 years old. Span. J. Psychol. 14, 478–485. doi: 10.5209/rev_SJOP.2011.v14.n1.43
Mewton, L., Kessler, R. C., Slade, T., Hobbs, M. J., Brownhill, L., Birrell, L., et al. (2016). The psychometric properties of the kessler psychological distress scale (K6) in a general population sample of adolescents. Psychol. Assess. 28, 1232–1242. doi: 10.1037/pas0000239
Mogle, J. (2015). “Internet-based assessment,” in The Encyclopedia of Adulthood and Aging, ed S. Krauss (Hoboken, NJ: John Wiley & Sons), 1–5. doi: 10.1002/9781118521373.wbeaa172
Muñiz, J., and Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test [Ten steps for test development]. Psicothema. 31, 7–16. doi: 10.7334/psicothema2018.291
Muris, P., Merckelbach, H., Mayer, B., van Brakel, A., Thissen, S., Moulaert, V., et al. (1998). The screen for child anxiety related emotional disorders (SCARED) and traditional childhood anxiety measures. J. Behav. Ther. Exp. Psychiatry 29, 327–339. doi: 10.1016/S0005-7916(98)00023-8
Muris, P., Simon, E., Lijphart, H., Bos, A., Hale, W., and Schmeitz, K. (2017). The youth anxiety measure for DSM-5 (YAM-5): development and first psychometric evidence of a new scale for assessing anxiety disorders symptoms of children and adolescents. Child Psychiatry Hum. Dev. 48, 1–17. doi: 10.1007/s10578-016-0648-1
Orgiles, M., Fernandez-Martinez, I., Guillen-Riquelme, A., Espada, J. P., and Essau, C. A. (2016). A systematic review of the factor structure and reliability of the spence children's anxiety scale. J. Affect. Disord. 190, 333–340. doi: 10.1016/j.jad.2015.09.055
Orgiles, M., Mendez, X., Spence, S. H., Huedo-Medina, T. B., and Espada, J. P. (2012). Spanish validation of the spence children's anxiety scale. Child Psychiatry Hum. Dev. 43, 271–281. doi: 10.1007/s10578-011-0265-y
Piqueras, J. A., Garcia-Olcina, M., Rivera-Riquelme, M., and Pineda, D. (2020). Evidencias de validez diagnóstica de la escala detectaweb-malestar [Evidence of diagnostic validity of the detectaweb-distress scale]. J. Psychopathol. Clin. Psychol. Rev. Psicopatol. Psicol. Clín. 25, 159–173. doi: 10.5944/rppc.28931
Piqueras, J. A., Garcia-Olcina, M., Rivera-Riquelme, M., Rodriguez-Jimenez, T., Martinez-Gonzalez, A. E., and Cuijpers, P. (2017a). detectaweb project: study protocol of a web-based detection of mental health of children and adolescents. BMJ Open 7:e017218. doi: 10.1136/bmjopen-2017-017218
Piqueras, J. A., Martín-Vivar, M., Sandin, B., San Luis, C., and Pineda, D. (2017b). The revised child anxiety and depression scale: a systematic review and reliability generalization meta-analysis. J. Affect. Disord. 218, 153–169. doi: 10.1016/j.jad.2017.04.022
Piqueras, J. A., Rodriguez-Jimenez, T., Marzo, J. C., Rivera-Riquelme, M., Martinez-Gonzalez, A. E., Falco, R., et al. (2019). Social emotional health survey-secondary (SEHS-S): a universal screening measure of social-emotional strengths for spanish-speaking adolescents. Int. J. Environ. Res. Public Health. 16:4982. doi: 10.3390/ijerph16244982
Polanczyk, G. V., Salum, G. A., Sugaya, L. S., Caye, A., and Rohde, L. A. (2015). Annual research review: A meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J. Child Psychol Psychiatry Allied Discip. 56, 345–365. doi: 10.1111/jcpp.12381
Radloff, L. S. (1977). The CES-D scale: a self-report depression scale for research in the general population. Appl. Psychol. Meas. 1, 385–401. doi: 10.1177/014662167700100306
Reynolds, C. R., and Kamphaus, R. W. (2004). Behavior Assessment System for Children, 2nd Edn. Bloomington, MN: Pearson Assessments.
Reynolds, W. M. (1986). A model for the screening and identification of depressed children and adolescents in school settings. Prof. School Psychol. 1, 117–129. doi: 10.1037/h0090504
Rivera-Riquelme, M., Piqueras, J. A., and Cuijpers, P. (2019). The revised mental health inventory-5 (MHI-5) as an ultra-brief screening measure of bidimensional mental health in children and adolescents. Psychiatry Res. 274, 247–253. doi: 10.1016/j.psychres.2019.02.045
Sandin, B., Chorot, P., Valiente, R. M., and Chorpita, B. F. (2010). Development of a 30-item version of the revised child anxiety and depression scale. Rev. Psicopatol. Psicol. Clín. 15, 165–178. doi: 10.5944/rppc.vol.15.num.3.2010.4095
Sandin, B., Valiente, R. M., and Chorot, P. (2009). RCADS: evaluación de los síntomas de los trastornos de ansiedad y depresión en niños y adolescentes Rev. Psicopatol. Psicol. Clín. 14, 193–206. doi: 10.5944/rppc.vol.14.num.3.2009.4078
Schermelleh-Engel, K., Moosbrugger, H., and Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods Psychol. Res. Online 8, 23–74.
Schumacker, R. E., and Lomax, R. G. (2004). A Beginner's Guide to Structural Equation Modeling. Mahwah, NJ: Routledge Taylor and Francis Group. doi: 10.4324/9781410610904
Southam-Gerow, M. A., and Chorpita, B. F. (2007). “Fears and Anxieties,” in Assessment of Childhood Disorders, eds E. J. Mash, and R. A. Barkley (New York, NY: Guildford), 271–335.
Spence, S. H. (1997). Structure of anxiety symptoms among children: a confirmatory factor- analytic study. J. Abnorm. Psychol. 106, 280–297. doi: 10.1037/0021-843X.106.2.280
Spence, S. H. (2018). Assessing anxiety disorders in children and adolescents. Child Adolesc. Ment. Health 23, 266–282. doi: 10.1111/camh.12251
Stiffler, M., and Dever, B. (2015). “Instrumentation for mental health screening,” in Mental health Screening at School. Contemporary Issues in Psychological Assessment (Switzerland: Springer International Publishing), 39–75. doi: 10.1007/978-3-319-19171-3
Stockings, E., Degenhardt, L., Lee, Y. Y., Mihalopoulos, C., Liu, A., Hobbs, M., et al. (2015). Symptom screening scales for detecting major depressive disorder in children and adolescents: a systematic review and meta-analysis of reliability, validity and diagnostic utility. J. Affect. Disord. 174, 447–463. doi: 10.1016/j.jad.2014.11.061
Susman, E. J., and Dorn, L. D. (2009). “Puberty: it's role in development,” in Handbook of Adolescent Psychology, 3rd Edn., eds R. M. Lerner, and L. Steinberg (Hoboken, NJ: John Wiley and Sons), 116–151. doi: 10.1002/9780470479193.adlpsy001006
Tran, T. D., Kaligis, F., Wiguna, T., Willenberg, L., Nguyen, H. T. M., Luchters, S., et al. (2019). Screening for depressive and anxiety disorders among adolescents in Indonesia: formal validation of the centre for epidemiologic studies depression scale–revised and the kessler psychological distress scale. J. Affect. Disord. 246, 189–194. doi: 10.1016/j.jad.2018.12.042
van Ballegooijen, W., Riper, H., Cuijpers, P., van Oppen, P., and Smit, J. H. (2016). Validation of online psychometric instruments for common mental health disorders: a systematic review. BMC Psychiatry 16:45. doi: 10.1186/s12888-016-0735-7
Veit, C. T., and Ware, J. E. (1983). The structure of psychological distress and well-being in general populations. J. Consult. Clin. Psychol. 51, 730–742. doi: 10.1037/0022-006X.51.5.730
Watson, D. (2005). Rethinking the mood and anxiety disorders: a quantitative hierarchical model for DSM-V. J. Abnorm. Psychol. 114, 522–536. doi: 10.1037/0021-843X.114.4.522
World Health Organization (2014a). Health for the World's Adolescents: A Second Chance in the Second Decade: Summary (No. WHO/FWC/MCA/14.05). Geneva: World Health Organization.
World Health Organization (2014b). Preventing Suicide: A Global Imperative. Geneva: World Health Organization.
Yule, W. (1997). Anxiety, Depression, and Posttraumatic Stress Disorder in Children., In I. Sclare (Ed.), Child Psychology Portfolio. Windsor: NFER-Nelson.
Keywords: emotional, distress, anxiety, depression, screener, children and adolescents, web-based, detectaweb-distress
Citation: Piqueras JA, Garcia-Olcina M, Rivera-Riquelme M, Martinez-Gonzalez AE and Cuijpers P (2021) DetectaWeb-Distress Scale: A Global and Multidimensional Web-Based Screener for Emotional Disorder Symptoms in Children and Adolescents. Front. Psychol. 12:627604. doi: 10.3389/fpsyg.2021.627604
Received: 09 November 2020; Accepted: 11 January 2021;
Published: 15 February 2021.
Edited by:
Francisco Javier Méndez, University of Murcia, SpainReviewed by:
Eduardo Fonseca-Pedrero, University of La Rioja, SpainMaría Paz García-Vera, Complutense University of Madrid, Spain
Josefa Canals, University of Rovira i Virgili, Spain
Copyright © 2021 Piqueras, Garcia-Olcina, Rivera-Riquelme, Martinez-Gonzalez and Cuijpers. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jose A. Piqueras, jpiqueras@umh.es