Hsin-Yi Wang
Cecilia Cheng- Department of Psychology, The University of Hong Kong, Pokfulam, Hong Kong
Gaming disorder was listed as a condition for further study in the 5th edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) in 2013, and measures of the disorder have mushroomed in the years since. The Gaming Disorder Test (GDT) was developed after gaming disorder was officially included in the 11th Revision of the International Classification of Diseases (ICD-11) in 2018. However, it remains unknown whether the GDT, which is based on the ICD-11 framework, is psychometrically similar to or different from the popular nine-item Internet Gaming Disorder Scale-Short Form (IGDS9-SF) based on the DSM-5 framework. To address this important but unexplored issue, the present study evaluated and compared the psychometric properties of the GDT and IGDS9-SF in a sample of 544 adult gamers (56.2% men; mean age = 28.8, SD = 8.55). The results revealed both measures to have good reliability, structural validity, and criterion validity, with the exception of one IGDS9-SF item with a low factor loading. Moreover, the IGDS9-SF exhibited scalar measurement invariance for gender and age but only partial metric invariance for employment status, whereas the GDT exhibited scalar measurement invariance for all three demographic characteristics. Finally, the GDT displayed incremental validity over the IGDS9-SF in explaining gaming time, but not social anxiety and depressive symptoms. This study thus contributes to the literature by comparing measures derived from distinct gaming disorder diagnostic frameworks empirically. Recommendations for the selection of gaming disorder measures by researchers and practitioners are discussed.
Introduction
Video gaming has become an integral part of life for many players, but gaming can become problematic if it interferes with psychosocial functioning [e.g., (1, 2)]. Reflecting widespread concerns over problematic gaming, Internet gaming disorder was listed as a condition for further study in the 5th edition of the Diagnostic and Statistical Manual of Mental Disorder [DSM-5; (3)]. The DSM-5 framework comprises nine criteria: 1) preoccupation with gaming; 2) withdrawal symptoms when gaming is not accessible; 3) increasing amounts of time spent on gaming; 4) unsuccessful attempts to control gaming; 5) loss of interest in other hobbies or activities; 6) continued excessive gaming despite knowledge of the undesirable consequences; 7) deceiving others regarding the amount of gaming; 8) use of gaming to escape from unpleasant moods; and 9) losing significant interpersonal relationships due to gaming. Although the DSM-5 labels the problem “Internet gaming disorder,” it states that the disorder also involves non-Internet or offline games (3). Since the DSM-5 framework's proposal, scholars have advocated for the establishment of standardized assessment tools based on the nine DSM-5 criteria (4, 5). In response, researchers have adopted the DSM-5 framework to develop measures of gaming disorder [e.g., (6)], with one of the most popular measure being the nine-item Internet Gaming Disorder Scale-Short Form [IGDS9-SF; (7)]. The IGDS9-SF has been widely adopted and validated in a range of cultural regions such as Australia, Hong Kong, Portugal, and Turkey [e.g., (8–11)].
More recently, gaming disorder was officially included as a mental health disorder in the 11th version of the International Classification of Diseases [ICD-11 (12)] to facilitate further research and the formulation of social policy. The ICD-11 framework consists of four criteria: (1) impaired control over gaming; (2) increasing priority given to gaming as gaming takes precedence over other interests; (3) the continuation of gaming despite knowledge of the undesirable consequences, and (4) the problematic gaming behavior has led to significant disruptions to major life domains (e.g., interpersonal, job/academic performance) that last for at least 12 months. The Gaming Disorder Test [GDT (2)] was developed based on this framework, with the measure validated in China and the United Kingdom.
Apparently, the two clusters of measures differ in certain ways because they are constructed from frameworks with a distinct set of criteria. For example, the more concise ICD-11 framework places greater emphasis on the functional impairment aspect of gaming disorder (13). For instance, past studies have identified several symptoms included in the ICD-11 framework (i.e., “loss of control” and “giving up other activities”) to make strong contribution in identifying gaming disorder [e.g., (13, 14)]. In contrast, the more comprehensive DSM-5 framework encompasses a wider variety of cognitive and behavioral manifestations of the disorder.
A recent study has tested validated measures constructed from both frameworks, and the findings showed that the two measures yielded comparable prevalence rates and similar patterns of associations with mental health indicators (15). The present study extended their work by adopting a more naunced approach to scrutinize whether there are differences in psychometric properties among measures derived from distinct frameworks. Specifically, this study compares several types of validity of the newly developed GDT (2) and the widely adopted IGDS9-SF (7). We conducted four tests for evaluating the fundamental psychometric properties of these two measures, including structural validity, criterion validity, concurrent validity, and reliability analysis.
We further scrutinized whether the GDT and IGDS9-SF have measurement invariance, which refers to the statistical property whereby respondents reporting the same scores tend to exhibit an identical level of the underlying trait (16). As gaming disorder measures are often used in large-scale population surveys with heterogeneous community samples [e.g., (17–19)], measurement invariance is essential for detecting the influence of demographic characteristics in between-group comparisons.
In this study, we examined the measurement invariance properties across three demographic characteristics1: (1) gender (men vs. women), (2) age (younger vs. older players), and (3) employment status (students, full-time workers, vs. non-full-time workers). Potential variations in gender and age were tested because studies have indicated that gaming disorder tends to be more prevalent among male players and younger players [e.g., (24–28)]. Potential variations in employment status were tested because studies have revealed that student players are more likely to be active gamers than those in employment, and the employed players are more likely than students to purchase in-game items (29).
Moreover, incremental validity is another important property, particularly for evaluating the performance of new assessment tools relative to existing ones [e.g., (30, 31)]. As the IGDS9-SF has consistently been shown to account for the variance in such mental health-related criterion measures as depressive and social anxiety symptoms [e.g., (32, 33)], as well as gaming time [e.g., (10, 34)], it is important to test whether the new GDT can explain the variance in these criterion measures beyond the contribution made by the existing IGDS9-SF. Such findings can inform researchers and practitioners the unique contribution of the newly developed GDT, and such information is useful for assisting their decisions regarding whether to include the new or existing measures in their protocol.
Materials and Methods
Participants and Procedures
Participants were recruited from Prolific Academic, because studies have reported this online participant pool to be viable and valid for academic research; moreover, the participants recruited from this platform were found to be more diverse and less dishonest compared to other similar online platforms (35, 36). Our compensation rate ($1 for 10 min) adhered to the regulations of Prolific Academic. Eligible participants were adults (i.e., aged over 18) who responded “yes” to the screening question “Have you played any video game in the past 12 months?” To ensure good data quality, the participants were screened based on their reputation on the survey platform (i.e., >95% approval rating). The data collection process was conducted via an online survey hosted on Qualtrics. The present protocol received prior ethical approval from the institutional review board of the authors' university.
We recruited 544 participants from 34 countries. Most were from Europe (41.3%) and North America (40.5%), while the remaining were from Asia (12.1%) and other regions (9.3%). More than half (56.2%) were men, and the average age of the sample was 28.8 (SD = 8.55, range = 18–62). Roughly one-third (32.2%) reported themselves as current students, 40.4% as full-time workers, and 27.4% as non-full-time workers (i.e., working part-time, unemployed, or working in an unpaid position). For the present sample, the self-reported weekly gaming time was 19.9 hours (SD = 14.9) for general gaming and 13.2 h for online gaming (SD = 12.8).
Measures
The Gaming Disorder Test [GDT (2)] was adopted to measure gaming disorder based on the ICD-11 framework. This scale includes four items to examine recurrent gaming over a 12-month period, regardless of the mode of gaming (e.g., online or offline) or the device used for playing (e.g., consoles, mobile devices, or personal computers). Participants were asked to rate all items on a 5-point Likert scale, ranging from 1 (never) to 5 (very often).
The nine-item Internet Gaming Disorder Scale-Short Form [IGDS9-SF; (7)] was selected to assess gaming disorder based on the DSM-5 framework. This instrument includes nine items that tap the severity of gaming disorder symptoms by evaluating gaming activities occurring over the past 12 months. Each item was measured on a 5-point Likert scale, ranging from 1 (never) to 5 (very often).
The short form of the Social Anxiety Interaction Scale [SIAS-6 (37)] was chosen to measure social anxiety symptoms. This scale adopts a unidimensional measurement approach to evaluate social apprehension based on six items. Participants were instructed to rate each item on a 5-point Likert scale, ranging from 1 (not at all characteristic or true of me) to 5 (extremely characteristic or true of me). The SIAS-6 was found to be reliable in this study (Cronbach's α = 0.88).
The short form of the Center for Epidemiological Studies Depression Scale [CES-D 10 (38)] was used to evaluate depressive symptoms. This unidimensional, non-clinical measure is comprised of 10 items that examined the frequency and duration of various symptoms of depression. Each item was measured in a 4-point Likert scale, ranging from 1 (rarely or none of the time) to 4 (most or all of the time). The CES-D had acceptable reliability in our study (Cronbach's α = 0.88).
The amount of gaming was assessed with two items asking how many hours per day, on average, participants played video games online and offline, respectively. Their weekly gaming time was calculated by multiplying the daily gaming time by 7 for each mode of video gaming.
Participants were also asked to report their age, gender, and employment status. For employment status, five options were provided: (1) student, (2) full-time worker, (3) part-time worker, (4) unemployed, and (5) not in paid work. To facilitate further analysis, the employment status is further recoded as a three-level construct, including student, full-time worker, and non-full-time worker, with the level of non-full-time worker comprising participants who were currently working part time, unemployed, and not in paid work.
Statistical Analysis
All statistical analyses were performed with R-Studio version 3.4.1 (RStudio Team, Boston, MA) and SPSS version 23 (IBM Corp., Armonk, NY). The internal consistency of the measures was assessed using both Cronbach's alpha and McDonald's omega. McDonald's omega was obtained from the psych package for R [version 2.0.9 (39)]. Associations among the study variables were examined using Pearson zero-order correlation analysis. Confirmatory factor analysis (CFA) and multigroup CFA were performed using the lavaan package for R [version 5.20 (40)].
Results
Psychometric Property Evaluation
Reliability and Structural Validity
Both the GDT and IGDS9-SF were found to be reliable (Cronbach's alphas = 0.86 and 0.82, and McDonald's omega = 0.86 and.83, respectively). CFA was conducted for assessing structural validity. To evaluate model fit, we adopted the widely adopted criteria: CFI > 0.095, TLI > 0.095, RMSEA < 0.06, SRMR < 0.08 (41). As the scale items for both measures have five response options, the diagonally weighted least-squares method (DWLS) was used to estimate the model because of its suitability for CFA with ordered categorical variables. In addition, the issues of non-normality were not pertinent for our analyses, because the DWLS method was utilized. For the GDT, the CFA model showed a good fit to the data, as indicated by all five model fit indices: χ2 = 5.646, CFI = 0.991, TLI = 0.992, RMSEA = 0.017, and SRMR = 0.016. Moreover, all of the factor loadings were statistically significant, and no factor loading for an individual item was below 0.70 (see the upper panel of Table 1). Hence, these findings provide robust support for the structural validity of the GDT.
Table 1. Descriptive statistics and factor loadings derived from confirmatory factor analytical model for the two gaming disorder measures.
For the IGDS9-SF, the unidimensional model structure had an acceptable fit to the data: χ2 = 71.223, CFI = 0.952, TLI = 0.954, RMSEA = 0.064, and SRMR = 0.060. All of the factor loadings were statistically significant; however, Item 8 (escape) had a factor loading below 0.50 (see the lower panel of Table 1). In addition, the mean score of Item 8 was notably higher than the rest of the items.
Criterion and Concurrent Validity
The criterion validity was first tested with correlation analysis. Both the GDT and IGDS9-SF had significant positive associations with social anxiety symptoms (GDT: r = 0.41; IGDS9-SF: r = 0.40) and depressive symptoms (GDT: r = 0.40; IGDS9-SF: r = 0.39). To address potential measurement errors, a CFA model was then constructed to further examine the criterion validity of the GDT and IGDS9-SF, respectively (see Figures 1, 2). Both models included three latent variables: gaming disorder, social anxiety symptoms, and depressive symptoms. The results were consistent with the correlation analysis. For the GDT, the model fit indices indicated a good fit with the data: χ2 = 362.096, CFI = 0.976, TLI = 0.960, RMSEA = 0.054, and SRMR = 0.062. For the IGDS9-SF, the model fit indices indicated an acceptable fit with the data: χ2 = 662.870, CFI = 0.958, TLI = 0.952, RMSEA = 0.060, and SRMR = 0.070.
Figure 1. Confirmatory factor analytical model testing the criterion validity of Gaming Disorder Test. GD, gaming disorder (assessed by GDT); SA, social anxiety symptoms; DEP, depressive symptoms. The figures inside the boxes represent the item numbers of the respective measures. All parameters are standardized.
Figure 2. Confirmatory factor analytical model testing the criterion validity of nine-item Internet Gaming Disorder Scale-Short Form. GD, gaming disorder (assessed by IGDS9-SF); SA, social anxiety symptoms; DEP, depressive symptoms. The figures inside the boxes represent the item numbers of the respective measures. All parameters are standardized.
Measurement Invariance
To evaluate measurement invariance, we performed multigroup CFA (16) to assess their measurement and scaling properties across three sample characteristics: gender, age, and employment status. This analysis involves evaluating progressively more constrained models to establish invariance on three levels. At the initial level, configural invariance analysis evaluates whether the number of factors and patterns of free factor loadings are equal across different demographic groups. Specifically, the configural invariance was tested by constraining the unidimensional factor structure to be invariant, but freely estimating the factor loadings and item intercepts. At the next level, metric invariance analysis imposes additional constraints that require the item factor loadings to be equal across demographic groups. At the final level, scalar invariance analysis holds the item intercepts on the latent variables to be equal across groups. When examining the differences among the increasingly more constrained models, relative changes in not only χ2 but also CFI and RMSEA were examined, because the chi-square difference test has been found oversensitive in measurement invariance analysis. A change ≤-0.01 in the CFI and a change greater than or equal to 0.015 in the RMSEA were chosen as the cut-offs for non-invariance [see (16, 42)].
GDT
The present findings constituted robust evidence of the GDT's scalar measurement invariance for the demographic characteristics of gender, age, and employment status (see the upper panel of Table 2). The cross-model differences evaluated with CFI (all <0.01), RMSEA (all <0.015), and chi-square differences (all non-significant) consistently indicated the invariance of the GDT with respect to the number of factors, item factor loadings, and item intercepts.
Table 2. Measurement invariance for gender, age, and employment status for two gaming disorder measures.
IGDS9-SF
Measurement invariance analysis showed the IGDS9-SF to have scalar invariance for the characteristics of gender and age (see the lower panel of Table 2), with such properties supported by the cross-model differences evaluated with multiple indices, including CFI (all <0.01), RMSEA (all <0.015), and chi-square differences (all non-significant).
For the characteristic of employment status, only configural invariance was supported by the model fit indices, and non-invariance was identified in the metric invariance level, as indicated by the increase in RMSEA (i.e., 0.016) and significant chi-square differences. We located the non-invariant parameters by examining the modification indices, with the factor loadings of Item 3 (tolerance) and Item 7 (deception) appearing to be non-invariant. Hence, a partial invariance model with the two non-invariant factor loadings left unconstrained was constructed. This model revealed an adequate and comparable model fit to that of the configural invariance model.
Incremental Validity
Hierarchical multiple regression analysis was performed to examine the additional contribution of the newly developed GDT over the existing widely used IGDS9-SF to explaining social anxiety symptoms, depressive symptoms, and gaming time. Following the conventional practice of incremental validity analysis, only the constructs identified as significantly associated with the two gaming disorder measures were included in the multiple regression analysis (43). Hence, prior to incremental validity testing, bivariate zero-order correlation analysis was conducted to check whether the proposed constructs should be included in this particular analysis.
Social Anxiety and Depressive Symptoms
As shown in Table 3, after controlling for the existing IGDS9-SF measure, the newer GDT measure was unable to explain a significant proportion of the additional variance for social anxiety and depressive symptoms. Thus, the results failed to support the incremental validity of the GDT for predicting these two mental health-related criterion measures.
Table 3. Hierarchical multiple regression for evaluating incremental validity of Gaming Disorder Test.
Gaming Time
Both the GDT (r = 0.30) and IGDS9-SF (r = 0.25) had significant positive associations with weekly general gaming time. As shown in Table 3, the GDT accounted for a significantly higher proportion of the variance after controlling for the IGDS9-SF. Similar to weekly general gaming time, both the GDT (r = 0.29) and IGDS9-SF (r = 0.22) were positively associated with weekly online gaming time. Hierarchical multiple regression analysis revealed that after controlling for the IGDS9-SF, the GDT accounted for a significant proportion of the additional variance in this variable. Taken together, these results indicate the incremental validity of the GDT over the IGDS9-SF in explaining both weekly general and weekly online gaming time.
Discussion
We conducted a psychometric investigation of the GDT developed based on the ICD-11 framework (2) and the IGDS9-SF developed based on the DSM-5 framework (7). For the GDT, the present findings show that it is a reliable measure. The findings also provide support for its structural validity in terms of the excellent model fit across all fit indices, indicating that the unidimensional structure as described in the ICD-11 framework reflects the optimal factor structure for the construct of gaming disorder. This factor structure not only coincides with that proposed in the initial development of GDT (2), but also corroborates with various past attempts to develop and validate measures derived from the DSM-5 framework [e.g., IGDT-10 by Király et al. (44); CIGDS by Sigerson et al. (20)].
For the IGDS9-SF, it is also found to be reliable. In addition, the unidimensionality of such measure has also been supported by most of the fit indices; but Item 8 of this measure— “Do you play in order to temporarily escape or relieve a negative mood (e.g., helplessness, guilt, anxiety)?—had a particularly weak factor loading. Moreover, the average score of this item is found considerably higher than the rest of the measure in the present study. It is noteworthy that using games to “escape” has been frequently investigated as a type of gaming motivation, namely, the escapism motivation [e.g., (45, 46)]. Studies have shown that escapism motivation is reported by not only players with abundant gaming disorder symptoms [e.g., (47, 48)] but also those who do not endorse symptoms of such disorder [see review by Lee (49)]. Hence, our findings imply that this particular IGDS9-SF item may not be sufficiently sensitive to differentiate between individuals reporting numerous gaming disorder symptoms and those reporting few such symptoms.
Second, evidence of configural, metric, and scalar invariance was found for GDT concerning all three demographic characteristics (i.e., gender, age, and employment status), whereas for the IGDS9-SF, such evidence was found for gender and age alone, with the results only partially supporting metric invariance for employment status. More specifically, the findings revealed that Item 3 (tolerance) and Item 7 (deception) had weaker factor loadings in the student sample than in the employed sample.
Such variations may be attributable to the different interpretations of these items regarding in-game investment. Conventionally, gaming time has been conceptualized as the primary form of such investment [e.g., (50, 51)]. For instance, both Item 3 (tolerance) and Item 7 (deception) were evaluated based on the “amount of time” spent on gaming. However, some scholars maintain that in-game monetary expenditure should also be acknowledged as a form of investment (52–54). Thus, it is important to consider how monetary and non-monetary (e.g., time) investment may differentially influence distinct groups of players. For instance, long gaming time has been identified as a concern for employed players, as it can interfere with their work performance and even result in disciplinary action [e.g., (55, 56)]. In contrast, recent findings indicate that games encouraging high in-game expenditure tend to jeopardize adolescents, who are not financially independent, and may thus have to deceive family members or friends to cover that expenditure (57). Hence, the undesirable outcomes associated with a greater amount of time spent on gaming may be perceived differently by student players and employed players.
Lastly, our analysis produced mixed findings concerning the incremental validity of the GDT over the IGDS9-SF. For instance, the results showed the GDT to have incremental validity over the IGDS9-SF in predicting both general and online gaming time, but no such validity was found for either depressive or social anxiety symptoms. Such incremental validity concerning gaming time may reflect potential differences between the more concise ICD-11 criteria and more comprehensive DSM-5 criteria. In addition to the criterion of “escape,” several additional criteria in the DSM-5 framework, including withdrawal and tolerance symptoms, have also been scrutinized concerning their relevance to gaming behavior [e.g., (50, 58)]. A recent review (59) maintained that gaming disorder may arise without the presence of withdrawal symptoms, and the current conceptualization of tolerance symptoms has also been criticized for its direct adaptation from substance addiction criteria without considering the nature of gaming activity (51). Thus, such criteria may partially weaken the explanatory power of the IGDS9-SF for time spent on gaming.
Implications for Future Research and Practice
The findings of this study have several implications for future research on gaming disorder. As the literature contains a number of gaming disorder measures developed under the DSM-5 framework (e.g., IGDS9-SF, IGDT-10), whereas a more recent measure was developed under the new ICD-11 framework (i.e., GDT), researchers may wonder which should be the most appropriate measure for their studies. As a guide for researchers, our study identifies two major issues to consider when deciding which measure to use to assess gaming disorder.
A major factor to consider is the length of a gaming disorder measure (6). As impulsivity has frequently been identified as a risk factor for such disorder [e.g., (60, 61)], brief assessment tools are preferable for use among individuals with high levels of impulsivity (2). Moreover, brief measures are also preferred for use in online surveys, because a shorter survey length has been associated with lower drop-out rates (62). Hence, the shorter GDT, at just four items, may be a more suitable choice to meet these purposes. Compared to the IGDS9-SF whose length is twice longer, the GDT provides an equally good or even better explanation of the data, as indicated by its incremental validity over the IGDS9-SF in accounting for variances in gaming time, as well as its scalar invariance property concerning employment status that is not found for the IGDS9-SF. Hence, researchers requiring methods with a short protocol (e.g., telephone survey) will find the GDT to be a brief, valid assessment tool that can be easily administered to a heterogeneous sample.
Another major factor to consider is the comprehensiveness of a measure, that is, whether the measure sufficiently examines all important domains of gaming disorder (63). Compared to the GDT, the IGDS9-SF offers a more comprehensive examination of gaming disorder based on the nine criteria. The comprehensive scope of the IGDS9-SF makes it potentially more appropriate for use in studies examining multidimensional gaming-related constructs that evaluate various aspects of gaming (e.g., gaming motivation, player identification). However, in our psychometric analyses, we identified several IGDS9-SF items that require further scrutiny. For example, similar to previous studies [e.g., (26, 64)], we found the item assessing the “escape” criteria to have poor discriminatory power. Thus, future inquiries should evaluate whether this item is a precise indicator of gaming disorder, or whether it simply reflects a high, but not problematic, level of engagement in gaming. Such a conceptual distinction is necessary because excessive gaming and high engagement in gaming have been found to represent diverse constructs, with the former explaining a large portion of variance in socially undesirable personality characteristics (e.g., introversion, neuroticism) while no such findings are yielded for the latter (65). It is also noteworthy that the two IGDS9-SF items concerning tolerance and deception yielded a poorer estimation of gaming disorder severity among student players than employed players. Hence, researchers should take respondents' employment status into consideration when using the IGDS9-SF in future studies.
A third major factor to consider is item wording. Although several items appear in both the GDT and IGDS9-SF, it is necessary to scrutinize the differences regarding the wording of this common set of items. For instance, one item included in the IGDS9-SF has stronger emphasis on the interpersonal difficulties of players compared to that of the GDT. Specifically, Item 2 of the GDT reads: “I have continued gaming despite the occurrence of negative consequences,” whereas the corresponding item in the IGDS9-SF (Item 6) reads: “Have you continued your gaming activity despite knowing it was causing problems between you and other people?” Hence, recent experiences of interpersonal problems are necessary for the item to be endorsed for the IGDS9-SF, but the item of GDT is endorsed for respondents who have recent experiences of problems in any life domains (e.g., work/studies, health, and interpersonal relations). However, it is likely that the severity of gaming disorder for players with a sparse social network may be underestimated because they tend to have fewer undesirable interpersonal experiences, compared to their counterparts having a dense social network in which interpersonal conflicts are more likely to occur [e.g., (66, 67)]. Thus, researchers should also consider the implication of this wording choice when selecting measures for assessing gaming disorder.
Furthermore, the present findings also have implications for practitioners. Gaming disorder has recently been classified as one of the “disorders due to addictive behaviors” in the ICD-11. Healthcare professionals may utilize the diagnostic criteria and codes outlined in either the ICD-11 or DSM-5 for assessing symptoms of this emergent psychological problem. The present study provides empirical support for both the GDT and IGDS9-SF as valid measures for symptom assessment. However, the IGDS9-SF is preferred for screening clients who are at-risk for gaming disorder, because an optimal cut-off value has been identified for this measure with high sensitivity and specificity for distinguishing between gamers clinically diagnosed as having gaming disorder and those without this diagnosis (68). As GDT is a newer measure of gaming disorder, sensitivity, and specificity tests should be conducted among clinical samples to obtain a clinically useful cut-off for screening at-risk clients.
Research Caveats
Several potential limitations of this study should be noted. First, the study evaluated the utility of both the GDT and IGDS9-SF for use with non-clinical samples. Accordingly, its findings may not be generalizable to clinical populations, such as individuals diagnosed with clinical depression. Future studies should further validate the GDT with clinical samples to allow comparisons to be made between clinical and non-clinical samples exhibiting gaming disorder symptoms.
Second, our participants were recruited primarily from North America and Europe, and hence the results may not be generalizable to other cultural regions, including East Asia and South America. Given that gaming disorder is a global concern (1), further effort should be made to translate the new and promising GDT into various languages for psychometric validation. Previous multinational comparisons have identified considerable differences in thinking and behavior among the inhabitants of diverse cultural regions (69), and the measurement invariance of cultural region should be established to allow more extensive cross-cultural comparisons of the prevalence of gaming disorder.
Lastly, the present study employs the IGDS9-SF as the assessment tool developed under the DSM-5 framework. It is important to acknowledge that there are at least six other assessment tools that have been developed based on this framework, and a recent review has documented notable differences among these instruments regarding the number of items (ranging from 9 and 27 items), response format, and psychometric properties (6). Hence, although we aimed to evaluate and compare the psychometric properties of the GDT (derived from the ICD-11 framework) with the IGDS9-SF (derived from the DSM-5 framework), researchers should be cautious in generalizing the present findings to other assessment tools developed under the DSM-5 framework.
Conclusion
This study provides the first psychometric comparison of the new GDT and the popular IGDS9-SF. Both measures are found to be valid, reliable tools to assess gaming disorder; however, some notable differences have been identified regarding the properties of measurement invariance. In addition, the GDT demonstrates incremental validity over the IGDS9-SF in explaining gaming time. Based on these findings, we advise researchers to consider the factors of brevity and comprehensiveness when choosing an appropriate measure of gaming disorder.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics Statement
The studies involving human participants were reviewed and approved by University of Hong Kong's Human Research Ethics Committee. The participants provided online informed consent to participate in this study.
Author Contributions
The study was designed by authors CC and H-YW. H-YW coordinated the data collection process, performed all the statistical analyses, and completed the first draft of the manuscript. CC contributed to the editing and revision of the manuscript. All authors contributed to the article and approved the submitted version.
Funding
This study was funded by Hong Kong Research Grants Council's General Research Fund (17400714), and the University of Hong Kong's Seed Fund for Incubating Group-based Collaborative Research Projects (102009405) and Seed Fund for Basic Research (201711159216).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Footnotes
1. ^The methods for testing various types of measurement invariance were commonly adopted in previous studies [e.g., (20–23)].
References
1. Cheng C, Cheung MW-L, Wang H-y. Multinational comparison of internet gaming disorder and psychosocial problems versus well-being: meta-analysis of 20 countries. Comput Hum Behav. (2018) 88:153–68. doi: 10.1016/j.chb.2018.06.033
2. Pontes HM, Schivinski B, Sindermann C, Li M, Becker B, Zhou M, et al. Measurement and conceptualization of gaming disorder according to the World Health Organization framework: the development of the gaming disorder test. Int J Ment Health Addict. (2019). doi: 10.1007/s11469-019-00088-z
3. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Washington, DC: American Psychiatric Association (2013).
4. Griffiths M, King D, Demetrovics Z. DSM-5 internet gaming disorder needs a unified approach to assessment. Neuropsychiatry. (2014) 4:1–4. doi: 10.2217/npy.13.82
5. Király O, Griffiths MD, Demetrovics Z. Internet gaming disorder and the DSM-5: conceptualization, debates, and controversies. Cur Addict Rep. (2015) 2:254–62. doi: 10.1007/s40429-015-0066-7
6. Bernaldo-de-Quirós M, Labrador-Méndez M, Sánchez-Iglesias I, Labrador F. Measurement instruments of online gaming disorder in adolescents and young people according to DSM-5 criteria: a systematic review. Adicciones. (2019) 32:291–302. doi: 10.20882/adicciones.1277
7. Pontes HM, Griffiths MD. Measuring DSM-5 internet gaming disorder: Development and validation of a short psychometric scale. Comput Hum Behav. (2015) 45:137–43. doi: 10.1016/j.chb.2014.12.006
8. Evren C, Dalbudak E, Topcu M, Kutlu N, Evren B, Pontes HM. Psychometric validation of the Turkish nine-item internet gaming disorder scale-short form (IGDS9-SF). Psychiatry Res. (2018) 265:349–54. doi: 10.1016/j.psychres.2018.05.002
9. Leung H, Pakpour AH, Strong C, Lin Y-C, Tsai M-C, Griffiths MD, et al. Measurement invariance across young adults from Hong Kong and Taiwan among three internet-related addiction scales: Bergen social media addiction scale (BSMAS), smartphone application-based addiction scale (SABAS), and internet gaming disorder scale-short form (IGDS-SF9) (Study Part A). Addict Behav. (2019) 101:105969. doi: 10.1016/j.addbeh.2019.04.027
10. Pontes HM, Griffiths MD. Portuguese validation of the internet gaming disorder scale-short-form. Cyberpsychol Behav Soc Netw. (2016) 19:288–93. doi: 10.1089/cyber.2015.0605
11. Stavropoulos V, Beard C, Griffiths MD, Buleigh T, Gomez R, Pontes HM. Measurement invariance of the internet gaming disorder scale-short-form (IGDS9-SF) between Australia, the USA, and the UK. Int J Ment Health Addict. (2018) 16:377–92. doi: 10.1007/s11469-017-9786-3
12. World Health Organization. (2018). Gaming Disorder. Retrieved from http://www.who.int/features/qa/gaming-disorder/en/
13. Jo YS, Bhang SY, Choi JS, Lee HK, Lee SY, Kweon Y-S. Clinical characteristics of diagnosis for internet gaming disorder: comparison of DSM-5 IGD and ICD-11 GD diagnosis. J Clin Med. (2019) 8:945. doi: 10.3390/jcm8070945
14. Schivinski B, Brzozowska-Woś M, Buchanan EM, Griffiths MD, Pontes HM. Psychometric assessment of the internet gaming disorder diagnostic criteria: an item response theory study. Addict Behav Rep. (2018) 8:176–84. doi: 10.1016/j.abrep.2018.06.004
15. Montag C, Schivinski B, Sariyska R, Kannen C, Demetrovics Z, Pontes HM. Psychopathological symptoms and gaming motives in disordered gaming-A psychometric comparison between the WHO and APA diagnostic frameworks. J Clin Med. (2019) 8:1691. doi: 10.3390/jcm8101691
16. Cheung GW, Rensvold RB. Evaluating goodness-of-fit indexes for testing measurement invariance. Struct Equ Model. (2002) 9:233–55. doi: 10.1207/S15328007SEM0902_5
17. Deleuze J, Rochat L, Romo L, Van der Linden M, Achab S, Thorens G, et al. Prevalence and characteristics of addictive behaviors in a community sample: a latent class analysis. Addict Behav Rep. (2015) 1:49–56. doi: 10.1016/j.abrep.2015.04.001
18. Festl R, Scharkow M, Quandt T. Problematic computer game use among adolescents, younger and older adults. Addiction. (2013) 108:592–9. doi: 10.1111/add.12016
19. Pearcy BT, Roberts LD, McEvoy PM. Psychometric testing of the personal internet gaming disorder evaluation-9: a new measure designed to assess internet gaming disorder. Cyberpsychol Behav Soc Netw. (2016) 19:335–41. doi: 10.1089/cyber.2015.0534
20. Sigerson L, Li AY-L, Cheung MW-L, Luk JW, Cheng C. Psychometric properties of the Chinese Internet Gaming Disorder Scale. Addict Behav. (2017) 74:20–6. doi: 10.1016/j.addbeh.2017.05.031
21. Burduli E, Barbosa-Leiker C, Fleming S, Hollins Martin CJ, Martin CR. Cross-cultural invariance of the birth satisfaction scale-revised (BSS-R): comparing UK and US samples. J Reprod Infant Psychol. (2017) 35:248–60. doi: 10.1080/02646838.2017.1310374
22. Gardner KJ, Qualter P. Factor structure, measurement invariance and structural invariance of the MSCEIT V2.0. Pers Individ Differ. (2011) 51:492–6. doi: 10.1016/j.paid.2011.05.004
23. González-Blanch C, Medrano LA, Muñoz-Navarro R, Ruíz-Rodríguez P, Moriana JA, Limonero JT, et al. Factor structure and measurement invariance across various demographic groups and over time for the PHQ-9 in primary care patients in Spain. PLoS One. (2018) 13:e0193356. doi: 10.1371/journal.pone.0193356
24. Dauriat FZ, Zermatten A, Billieux J, Thorens G, Bondolfi G, Zullino D, et al. Motivations to play specifically predict excessive involvement in massively multiplayer online role-playing games: evidence from an online survey. Eur Addict Res. (2011) 17:185–9. doi: 10.1159/000326070
25. Müller KW, Beutel M, Egloff B, Wölfling K. Investigating risk factors for internet gaming disorder: a comparison of patients with addictive gaming, pathological gamblers and healthy controls regarding the big five personality traits. Eur Addict Res. (2014) 20:129–36. doi: 10.1159/000355832
26. Rehbein F, Kliem S, Baier D, Mößle T, Petry NM. Prevalence of internet gaming disorder in German adolescents: diagnostic contribution of the nine DSM-5 criteria in a state-wide representative sample. Addiction. (2015) 110:842–51. doi: 10.1111/add.12849
27. Wang H-Y, Cheng C. New perspectives on the prevalence and associated factors of gaming disorder in Hong Kong community adults: A generational approach. Comput Hum Behav. (2021) 114:106574. doi: 10.1016/j.chb.2020.106574
28. Wang H-Y, Sigerson L, Cheng C. Digital nativity and information technology addiction: Age cohort versus individual difference approaches. Comput Hum Behav. (2019) 90:1–9. doi: 10.1016/j.chb.2018.08.031
29. Wu AM, Chen JH, Tong K-K, Yu S, Lau JT. Prevalence and associated factors of internet gaming disorder among community dwelling adults in Macao, China. J Behav Addict. (2018) 7:62–9. doi: 10.1556/2006.7.2018.12
30. Haynes SN, Lench HC. Incremental validity of new clinical assessment measures. Psychol Assess. (2003) 15:456–66. doi: 10.1037/1040-3590.15.4.456
31. Sigerson L, Cheng C. Scales for measuring user engagement with social network sites: a systematic review of psychometric properties. Comput Hum Behav. (2018) 83:87–105. doi: 10.1016/j.chb.2018.01.023
32. Burleigh TL, Stavropoulos V, Liew LW, Adams BL, Griffiths MD. Depression, internet gaming disorder, and the moderating effect of the gamer-avatar relationship: an exploratory longitudinal study. Int J Ment Health Addict. (2018) 16:102–24. doi: 10.1007/s11469-017-9806-3
33. Sarda E, Bègue L, Bry C, Gentile D. Internet gaming disorder and well-being: a scale validation. Cyberpsychol Behav Soc Netw. (2016) 19:674–9. doi: 10.1089/cyber.2016.0286
34. Laconi S, Pirès S, Chabrol H. Internet gaming disorder, motives, game genres and psychopathology. Comput Hum Behav. (2017) 75:652–9. doi: 10.1016/j.chb.2017.06.012
35. Palan S, Schitter C. Prolific.ac-A subject pool for online experiments. J Behav Exp Finance. (2018) 17:22–7. doi: 10.1016/j.jbef.2017.12.004
36. Peer E, Brandimarte L, Samat S, Acquisti A. Beyond the Turk: alternative platforms for crowdsourcing behavioral research. J Exp Soc Psychol. (2017) 70:153–63. doi: 10.1016/j.jesp.2017.01.006
37. Peters L, Sunderland M, Andrews G, Rapee RM, Mattick RP. Development of a short form Social Interaction Anxiety (SIAS) and Social Phobia Scale (SPS) using nonparametric item response theory: the SIAS-6 and the SPS-6. Psychol Assess. (2012) 24:66–76. doi: 10.1037/a0024544
38. Cole JC, Rabin AS, Smith TL, Kaufman AS. Development and validation of a Rasch-derived CES-D short form. Psychol Assess. (2004) 16:360–72. doi: 10.1037/1040-3590.16.4.360
39. Revelle W. psych: Procedures for Psychological, Psychometric, and Personality Research: Northwestern University, Evanston, IL. R package Version 2.0.9 (2020). Available online at: https://CRAN.R-project.org/package=psych
40. Rosseel Y. lavaan: An R package for structural equation modeling. J Stat Softw. (2012) 48:1–36. doi: 10.18637/jss.v048.i02
41. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model. (1999) 6:1–55. doi: 10.1080/10705519909540118
42. Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Model. (2007) 14:464–504. doi: 10.1080/10705510701301834
43. Petrides KV, Pérez-González JC, Furnham A. On the criterion and incremental validity of trait emotional intelligence. Cogn Emot. (2007) 21:26–55. doi: 10.1080/02699930601038912
44. Király O, Sleczka P, Pontes HM, Urbán R, Griffiths MD, Demetrovics Z. Validation of the ten-item internet gaming disorder test (IGDT-10) and evaluation of the nine DSM-5 internet gaming disorder criteria. Addict Behav. (2017) 64:253–60. doi: 10.1016/j.addbeh.2015.11.005
45. Demetrovics Z, Urbán R, Nagygyörgy K, Farkas J, Zilahy D, Mervó B, et al. Why do you play? The development of the motives for online gaming questionnaire (MOGQ). Behav Res Methods. (2011) 43, 814–25. doi: 10.3758/s13428-011-0091-y
46. Yee N. The demographics, motivations, and derived experiences of users of massively multi-user online graphical environments. Presence. (2006) 15:309–29. doi: 10.1162/pres.15.3.309
47. Bowditch L, Chapman J, Naweed A. Do coping strategies moderate the relationship between escapism and negative gaming outcomes in World of Warcraft (MMORPG) players? Comput Hum Behav. (2018) 86:69–76. doi: 10.1016/j.chb.2018.04.030
48. Hagström D, Kaldo V. Escapism among players of MMORPGs-Conceptual clarification, its relation to mental health factors, and development of a new measure. Cyberpsychol Behav Soc Netw. (2014) 17:19–25. doi: 10.1089/cyber.2012.0222
49. Lee Y. Exploring the Role of Escapism in the DSM-V Criteria for Internet Gaming Disorder: A Meta-Analytic Investigation. State University of New York at Albany, Albany, NY (2017) (Unpublished Doctoral Dissertation).
50. King DL, Herd MC, Delfabbro PH. Tolerance in internet gaming disorder: a need for increasing gaming time or something else? J Behav Addict. (2017) 6:525–33. doi: 10.1556/2006.6.2017.072
51. King DL, Herd MC, Delfabbro PH. Motivational components of tolerance in internet gaming disorder. Comput Hum Behav. (2018) 78:133–41. doi: 10.1016/j.chb.2017.09.023
52. Griffiths M. Is the buying of loot boxes in video games a form of gambling or gaming? Gaming Law Rev. (2018) 22:52–4. doi: 10.1089/glr2.2018.2216
53. Macey J, Hamari J. eSports, skins and loot boxes: participants, practices and problematic behaviour associated with emergent forms of gambling. New Media Soc. (2019) 21:20–41. doi: 10.1177/1461444818786216
54. Zendle D, Cairns P. Video game loot boxes are linked to problem gambling: results of a large-scale survey. PLoS One. (2018) 13:e0206767. doi: 10.1371/journal.pone.0206767
55. Korkeila J, Kaarlas S, Jääskeläinen M, Vahlberg T, Taiminen T. Attached to the web-harmful use of the internet and its correlates. Eur Psychiat. (2010) 25:236–41. doi: 10.1016/j.eurpsy.2009.02.008
56. Lim VK, Chen DJ. Cyberloafing at the workplace: gain or drain on work? Behav Inf Technol. (2012) 31:343–53. doi: 10.1080/01449290903353054
57. Dreier M, Wölfling K, Duven E, Giralt S, Beutel M, Müller K. Free-to-play: about addicted whales, at risk dolphins and healthy minnows. Monetarization design and internet gaming disorder. Addict Behav. (2017) 64:328–33. doi: 10.1016/j.addbeh.2016.03.008
58. Kuss DJ, Griffiths MD, Pontes HM. Chaos and confusion in DSM-5 diagnosis of internet gaming disorder: issues, concerns, and recommendations for clarity in the field. J Behav Addict. (2017) 6:103–9. doi: 10.1556/2006.5.2016.062
59. Kaptsis D, King DL, Delfabbro PH, Gradisar M. Withdrawal symptoms in internet gaming disorder: a systematic review. Clin Psychol Rev. (2016) 43:58–66. doi: 10.1016/j.cpr.2015.11.006
60. Choi S-W, Kim H, Kim G-Y, Jeon Y, Park S, Lee J-Y, et al. Similarities and differences among internet gaming disorder, gambling disorder and alcohol use disorder: A focus on impulsivity and compulsivity. J Behav Addict. (2014) 3:246–53. doi: 10.1556/JBA.3.2014.4.6
61. Ding W-N, Sun J-H, Sun Y-W, Chen X, Zhou Y, Zhuang Z-G, et al. Trait impulsivity and impaired prefrontal impulse inhibition function in adolescents with internet gaming addiction revealed by a Go/No-Go fMRI study. Behav Brain Funct. (2014) 10:20. doi: 10.1186/1744-9081-10-20
62. Hoerger M. Participant dropout as a function of survey length in internet-mediated university studies: implications for study design and voluntary participation in psychological research. Cyberpsychol Behav Soc Netw. (2010) 13:697–700. doi: 10.1089/cyber.2009.0445
63. Koronczai B, Urbán R, Kökönyei G, Paksi B, Papp K, Kun B, et al. Confirmation of the three-factor model of problematic internet use on off-line adolescent and adult samples. Cyberpsychol Behav Soc Netw. (2011) 14:657–64. doi: 10.1089/cyber.2010.0345
64. Ko C-H, Yen J-Y, Chen S-H, Wang P-W, Chen C-S, Yen C-F. Evaluation of the diagnostic criteria of internet gaming disorder in the DSM-5 among young adults in Taiwan. J Psychiatr Res. (2014) 53:103–10. doi: 10.1016/j.jpsychires.2014.02.008
65. Charlton JP, Danforth ID. Validating the distinction between computer addiction and engagement: online game playing and personality. Behav Inform Technol. (2010) 29:601–13. doi: 10.1080/01449290903401978
66. Lemmens JS, Valkenburg PM, Gentile DA. The internet gaming disorder scale. Psychol Assess. (2015) 27:567. doi: 10.1037/pas0000062
67. Wong PW, Li TM, Chan M, Law Y, Chau M, Cheng C, et al. The prevalence and correlates of severe social withdrawal (hikikomori) in Hong Kong: a cross-sectional telephone-based survey study. Int J Soc Psychiatry. (2015) 61:330–42. doi: 10.1177/0020764014543711
68. Severo RB, Barbosa APPN, Fouchy DRC, Coelho FMdC, Pinheiro RT, de Figueiredo VLM, et al. Development and psychometric validation of Internet Gaming Disorder Scale-Short-Form (IGDS9-SF) in a Brazilian sample. Addict Behav. (2020) 103:106191. doi: 10.1016/j.addbeh.2019.106191
Keywords: internet gaming disorder, online gaming, gaming addiction, behavioral addiction, scale validation, measurement invariance, psychometric comparison, psychometric properties
Citation: Wang H-Y and Cheng C (2020) Psychometric Evaluation and Comparison of Two Gaming Disorder Measures Derived From the DSM-5 and ICD-11 Frameworks. Front. Psychiatry 11:577366. doi: 10.3389/fpsyt.2020.577366
Received: 29 June 2020; Accepted: 27 November 2020;
Published: 17 December 2020.
Edited by:
Catherine Woodstock Striley, University of Florida, United StatesReviewed by:
Li-Zhuang Yang, Hefei Institutes of Physical Science (CAS), ChinaHalley M. Pontes, University of Tasmania, Australia
Copyright © 2020 Wang and Cheng. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Cecilia Cheng, ceci-cheng@hku.hk