The structure of dimensions of psychopathology in normative and clinical samples: Applying causal discovery to MMPI-2-RF scales to investigate clustering of psychopathology spectra and p-factors

We applied a Bayesian Constraint-based Causal Discovery method (BCCD) to examine the hierarchical structure of the Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF) Restructured Clinical (RC) scales. Two different general psychopathology super spectra (p-factor) scales were extracted from (1) all RC scales and (2) all RC scales except the RCd (Demoralization) scale. These p-factor scales were included in separate models to investigate the structure of dimensions of psychopathology in a normative (n = 3,242) and clinical (n = 2,466) sample, as well as the combined normative/clinical sample (N = 5,708), by applying the BCCD algorithm to obtain a data-driven reconstruction of the internal hierarchical structure of the MMPI-2-RF. Research on the underlying structure of the MMPI-2-RF has clinical relevance as well as conceptual relevance in the context of the HiTOP model. Results demonstrated that the syndromes measured with the RC-scales—in presence of a p-factor—cluster into six spectra: internalizing, disinhibited-externalizing, antagonistic-externalizing, thought disorder, detachment, and somatoform. These results may support a super spectrum construct, as it was necessary for obtaining a bottom-up reconstruction of this six-spectrum structure. We found support for superiority of a broad super spectrum with additional variance over and above demoralization, as it resulted in the clearest structure (i.e., clustering of the RC scales). Furthermore, our results indicate independent support for the bifactor structure model of psychopathology.

Introduction

The MMPI-2-RF and HiTOP

Contemporary diagnostic models of psychopathology are increasingly focused on quantitative approaches, mostly using factor analysis, to identify dimensionally operationalized psychopathology constructs and models (1–3), which has been particularly prominent in the context of personality pathology (4, 5). To define psychopathology beyond the predominantly categorical DSM-5 classifications (6), several dimensional models have been developed that rely on factor analysis to describe syndromes and symptoms in broader underlying variables (i.e., spectra). The Hierarchical Taxonomy of Psychopathology [HiTOP (3, 7–9)] is currently the most influential quantitative dimensional model of psychopathology. The HiTOP consortium proposed this data-driven hierarchical model, which incorporates various dimensions of psychopathology, including—from lower to higher level—symptoms, syndromes, subfactors, spectra, and a super spectrum (i.e., a general psychopathology-factor, p). The assessment of the HiTOP dimensions in clinical practice is work in progress (10), but as stated by Sellbom et al., at this time the Minnesota Multiphasic Personality Inventory-2-Restructured Form and 3 [MMPI-2-RF and MMPI-3; (11, 12)] are the most adequate and encompassing instruments to measure and operationalize the HiTOP model (13, 14).

The MMPI-2-RF measures dimensional and hierarchically organized psychopathology constructs on various levels that map onto the HiTOP structure (3, 14). The hierarchical structure of the MMPI-2-RF includes three higher-order (H-O) scales, nine restructured clinical (RC) scales (15), twenty-three specific problem (SP) scales, and five pathological personality dimensions [i.e., PSY-5-r; (16)]. Of particular interest in this study are the nine RC scales (see Table 1), that resemble the subfactors in HiTOP (see Figure 1). From these constructs, the Demoralization scale (RCd) stands out, due to its construction and high clinical relevance. This construct formerly represented shared variance of the Clinical scales of the MMPI-2 and reflects general distress and general dissatisfaction with life. RCd is associated with the internalizing spectrum (11, 17) and correlates highly with suicidality and a range of depressive symptoms (18, 19). The other eight RC scales measure syndromes of somatic complaints (RC1), low positive emotions (RC2), cynicism (RC3), antisocial behavior (RC4), ideas of persecution (RC6), dysfunctional negative emotions (RC7), aberrant experiences (RC8), and hypomanic activation (RC9) (11, 15, 19). The MMPI-2-RF is one of the most utilized clinical assessment instruments globally. Further research on the dimensional structure of the MMPI-2-RF has clinical relevance as well as conceptual ramifications in the context of the HiTOP model (3, 14).

TABLE 1

Table 1. Descriptives of RC-scales and H-O scales in the Combined Sample.

FIGURE 1

Figure 1. Hierarchical design structure of the MMPI-2-RF RC scales in accordance with the HiTOP model. Hierarchical design structure of the MMPI-2-RF, showing the six spectra and the corresponding RC-scales in accordance with HiTOP. Not depicted: the RC3 subfactor has associations with the RC2 subfactor (and Detachment spectrum) as well.

Similar to the MMPI-2-RF structure, HiTOP is a hierarchy of clusters of syndromes and symptoms, grouping lower level symptoms under broadly defined spectra (3). The internalizing spectrum encompasses symptoms of negative affectivity, anxiety, fear, sexual problems, eating problems, and depressed mood (3, 20). A related somatoform spectrum is considered to encompass symptoms of pain disorder, illness anxiety symptoms, as well as other somatic symptoms (13, 14, 21). Two externalizing spectra encompass disinhibited-externalizing and antagonistic acting-out behaviors (14). A fifth spectrum thought disorder (or thought dysfunction) encompasses psychotic, schizotypal and paranoid features (21, 22). Finally, the sixth spectrum is labeled the detachment spectrum which is defined by low positive emotionality and avoidant/schizoid and dependent personality symptomatology (3, 23).

HiTOP approach and advantages

Generally, the HiTOP model takes a quantitative, multidimensional, and hierarchical approach to the structure of psychopathology, which has several advantages over traditional, categorical classifications (3, 24). First, dimensional measures of psychopathology have evidenced superior reliability and validity to categorical diagnoses (25, 26). Also, dimensional profiles, such as those generated by the MMPI-2-RF, provide the opportunity for clinicians to reframe mental illness into strengths and vulnerabilities in various domains of functioning (24, 27). The hierarchical structure with broad dimensions at the head of the structure addresses the well-known problem of co-occurrence [i.e., comorbidity or symptom overlap; (28)] among diagnostic categories, whereas the small lower levels maintain the heterogeneity of symptoms and incorporate multifinality (7, 29). Hierarchical, dimensional measurement allows for comprehensive assessment of psychopathology; both adaptive and maladaptive personality dimensions (i.e., continua of personality, rather than clinical categories) fit well into the overall structure, given the numerous empirical links identified between personality traits and psychopathology (14, 30, 31). Finally, decreased use or disappearance of routinely assigned categorical, clinical “labels” may have a destigmatizing effect on the individual and their environment.

Despite these advantages, there are still several underexplained aspects of the HiTOP model, such as the lack of clarity about stability of the different levels of the model and the descriptive (as opposed to elucidative) nature of the model. In addition, the model is grounded in factor-analytic approaches, whereas several studies have indicated that psychopathology is better understood as networks of interconnected symptoms and traits (32–35). Moreover, some of the fundamental assumptions of HiTOP have recently been debated in the literature (8, 36). Nonetheless, the HiTOP structure has been studied across a variety of methods including self-report measures such as the MMPI (37–39), as well as interviews (40) and peer report studies (41). The dimensional nature of various types of psychopathology has been replicated across samples (42), cultures (21, 43, 44), and ages (45–47).

General psychopathology factor

Cross-correlations between spectra may point toward a general psychopathology construct or p-factor, a broad factor that underlies various phenotypes of psychopathology (48) and is therefore positioned as a super spectrum in the HiTOP model (3). This super spectrum or p-factor construct has been compared to the g-factor [i.e., general mental ability; (49)] and has gained momentum using latent trait analysis and network analysis (50). Particularly, a high level of p may be an indicator of a general risk for various types of psychopathology. A genotype and phenotype p-factor has been replicated as a general dimension of psychopathology across ages, methods and countries (1, 51, 52). For a critical evaluation of the p-factor literature to date the reader is referred to Watts and colleagues (53).

The true nature, reliable measurement, and direct clinical relevance of this construct has not been fully determined (53). An important topic for the current study is the discussion about different statistical approaches in the p-factor literature [i.e., bifactor vs. hierarchical models (54)]. The bifactor model is hypothesized to explain spectral clustering of syndromes and symptoms only through the p-factor construct, while the hierarchical model poses that the p-factor predicts spectra which in turn predict the underlying syndromes and symptoms (55). Recent evidence points toward a bifactor model, in which the p-factor construct was primarily associated with impulsivity, impairment and neuroticism constructs (56).

Replication of HiTOP's structure with p-factor

While most studies have evaluated the structural validity of the HiTOP model using traditional factor analytic methods, replicating the structure using advanced statistical methods without predefined assumptions about the structure would ensure the robustness and generalizability of the model (3, 57). Importantly, modern causal discovery methods are specifically designed for such hypothesis-free reconstructions by generating networks from data. Where classic methods of factor analysis compare different models by establishing a fit of the data and comparing likelihoods, causal discovery methods try to reconstruct the underlying generating network bottom-up in fully data driven approaches. With the ability to identify complex hierarchical networks in large data sets, they could provide a promising complementary tool for investigating the structure of the HiTOP model. Therefore, in the present study, we investigate the structure of the HiTOP model using a state-of-the-art causal discovery approach. We opt for the Bayesian Constraint-based Causal Discovery (BCCD) algorithm (58), because of its ability to handle latent confounders, and also provides an estimate of the reliability of each part of the network found. However, well-known alternatives like bootstrapped FCI (59) or HEJ (60) could also be applied. BCCD was previously used to successfully detect clustering patterns in the Wechsler Adult Intelligence Scale-Fourth Edition structure (61, 62).

The present study focuses on applying BCCD to MMPI-2-RF (11) RC scales (15) in both community and clinical samples. To investigate the existence, nature, and positioning of a general psychopathology factor, we extract two different p-factor scales from (1) all RC scale items and (2) all RC scale items except demoralization items. We examined the structure of the MMPI-2-RF scales in relation to these two operationalizations of the p-factor. The Demoralization construct was originally derived to capture a primarily-internalizing, overarching construct from the MMPI-2 Clinical scales. However, the p-factor is hypothesized to be associated not only with internalizing symptomatology but with all six spectra (i.e., internalizing, disinhibited-externalizing, antagonistic-externalizing, thought disorder, detachment, and somatoform) [e.g., (1, 56, 63)] and to be central at the top of the model as in a hierarchical model (14, 54, 55). Therefore, we compare the structure of both models with different p-factors (i.e., primarily defined by demoralization vs. broadly defined). If the HiTOP design and the hypothesized p-factors hold true, then we would expect clustering of RC-scales into six spectra (64). Through investigation of the cross links between subordinate dimensions of the model structures both with and without the general psychopathology dimensions, the present study can also shed new light on the discussion around bifactor vs. hierarchical structures of psychopathology [see Figure 2; (55)].

FIGURE 2

Figure 2. Explanatory models of the BCCD-method applied to dimensional MMPI-2-RF data. Higher order factor structure (left) revealed as clustering pattern through causal network analysis (bottom, right), provided “true” p-factor is sufficiently approximated. Without adequate p-factor proxy the network shows a high degree of latent interactions between the observed factors (in our case the separate RC-scales).

Work by Sellbom in particular has led to mapping of the RC scales onto the HiTOP model (13, 14), which has guided our hypotheses. We hypothesize clustering of the RC scales into 6 spectra, mapping onto the HiTOP spectral constructs, such that the constructs Demoralization (RCd) and Dysfunctional Negative Emotions (RC7) cluster into the internalizing spectrum. The detachment spectrum is proposed to be associated with Low Positive Emotions (RC2), which is also expected to show associations with the internalizing spectrum. The somatoform spectrum is in line with the construct measured by the Somatic Complaints construct (RC1). The disinhibited-externalizing spectrum is mapped onto the Antisocial Behavior construct (RC4), whereas the antagonistic-externalizing spectrum maps onto the Hypomanic Activation construct (RC9) and to a degree the Cynicism construct (RC3), the latter of which is expected to be associated with the thought disorder and internalizing spectrum as well (19). The sixth spectrum, thought disorder, is a clustering of the Ideas of Persecution (RC6) and Aberrant Experiences constructs (RC8) (13), with Cynicism (RC3) associations. We expect both p-factor scales (with and without demoralization) to further accentuate these existing syndrome clusters, with the p-factor scale without demoralization being the stronger overarching super spectrum by removing the most cross-variance between the clusters.

Method

Participants and sampling procedures

All participants completed the Dutch language version of the MMPI-2. Using the MMPI-2 booklet, MMPI-2-RF raw scores were generated, previously shown to be a valid method of conversion (11, 65). Combined gender norms T-scores were computed for all scales. For an overview of summary statistics of the MMPI-2-RF T-scores in the Combined sample, we refer to Table 1. All participants met the following validity inclusion criteria based on the Dutch language manual of the MMPI-2-RF: CNS < 14, VRIN-r/TRIN-r < = 80T, F-r < 120T, F-r < 100T, L-r < = 80T.

Normative subsample A

Participants in normative subsample A completed the MMPI-2 in 1992 in the context of standardization of the Dutch-language version of the MMPI-2 (66). This sample was stratified where needed to best represent normal distributions in the general Dutch population. All participants were financially compensated for their participation. MMPI-2 administration was computerized and participants who lacked the technical necessities were provided with the necessary equipment. For more information on the sampling procedures of normative subsample A, we refer to the manual of the first edition of the Dutch language version of the MMPI-2 (66). After removal of invalid participants and participants that also participated in the standardization of the Dutch language version of the MMPI-2 in 2012, the final 1992 sample included 1,157 participants (M_age = 43.6, SD_age = 14.3, 45.4% female).

Normative subsample B

Participants in normative subsample B were recruited in 2012 to update the norms of the Dutch language version of the MMPI-2. All participants were compensated for their participation, and administration was computerized (67). After removal of invalid participants, those who repeated assessments at both time points, and invalid survey responses, the final normative subsample B included 2,085 participants (M_age = 52.1, SD_age = 16.8, 48% female).

Normative sample

Normative subsamples A and B were merged into one Normative Sample for further analysis (n = 3,242, M_age = 49.1, SD_age = 16.4, 47.1% female).

Clinical sample

Independent from the normative sample administration, participants from six large, specialized inpatient and outpatient mental health facilities in the Netherlands filled out the MMPI-2 for the purpose of assessment and treatment planning. We removed participants with invalid survey responses, resulting in the final clinical sample (n = 2,466, M_age = 35.1, SD_age = 11.9, 42.3% female). These participants were included based on convenience sampling and inclusion followed their referral for psychological assessment and treatment planning because of multiple comorbid and severe mental disorders, including internalizing, externalizing and psychotic symptomatology.

Combined sample

The normative sample and clinical sample were merged into one Combined Sample (N = 5,708, M_age = 43, SD_age = 16.2, Range_age = 18–90, 45% female).

Sampling comparison

We compared the normative and clinical samples on age and gender. First, comparing the Normative Sample (M_age= 49.1, SD_age = 16.4) with the Clinical Sample (M_age = 35.1, SD_age = 11.9) on age, statistically significant differences between the means were found [t(5, 706) = 35.7, p < 0.001]. Second, Chi-square tests of independence were conducted to assess differences in gender between the two samples. A statistically significant difference [χ²(1, N = 5, 708) = 12.9, p < 0.001] was found between the Normative Sample (47.1% female) and Clinical Sample (42.3% female). We addressed these differences by running BCCD analyses with and without age and gender to assess whether the structure was consistent.

Informed consent

This study involves a unique secondary analysis of earlier published datasets (68–71). First, regarding the clinical sample, approval was granted by the Institutional Review Board of Reinier van Arkel Mental Health Institute [Date: April 2007/No.: ZO/AL/mg.04.2007; previously reported on by Van der Heijden et al. (69). Data were gathered for the purpose of psychological assessment and treatment planning. We extracted anonymous data from large datasets and this research was conducted in accordance with the guidelines for archival research from Reinier van Arkel Mental Health Institute based on Dutch law (i.c. WGBO art. 458). Second, for the normative sample, participants completed the MMPI-2 in context of the standardization process of the MMPI-2 which was done by Center Data in the Netherlands. Data collection was done entirely anonymously. Data from the Dutch normative sample were made available by the test publisher with consent by the participants [see (68)].

Analysis procedures

p-factor scales construction

In order to extract two general psychopathology (p-factor) scales, we first factor analyzed all RC scale items (i.e., 192 items) and then all RC scales items except items from RCd (i.e., 168 items) in the combined sample. We applied factor extraction on the combined sample (N = 5,708) to maximize the sample size and variability in constructing these scales. We ran exploratory factor analyses (EFAs) in SPSS version 28 (72) on both item pools (i.e., 192 and 168 items), fixing the factor extraction to 1 factor, with Maximum Likelihood extraction and 25 iterations. We extracted the 21 single highest loading items from the 192 and 168 item pools to construct two scales, as this number is the average number of items of the RC-scales.

Causal model analysis

We applied the Bayesian Constraint-based Causal Discovery algorithm (BCCD) on the available data to determine the hierarchical structure between the RC-scales, both with and without the two p-factor proxies. We only report models constructed using the combined sample data. We assessed five models (1) without p-factors and without RCd to assess free clustering of the RC scales, (2) with the p-factor that was based on eight RC scales, and without the Demoralization scale, (3) with the p-factor that was based on eight RC scales, and with the Demoralization scale, (4) with the p-factor based on all RC scale items (including RCd) with eight RC scales (not RCd), and (5) with the p-factor based on eight RC scales (without demoralization items) with nine RC scales (including RCd) as well as the three H-O scales to assess bifactor vs. hierarchical models. We reran these models for the different subsamples (normative/clinical) and with and without age and gender in the model.

Constraint-based causal discovery methods try to infer invariant features of the true underlying generating system from statistical patterns—conditional in- and dependence constraints—in the data. Under certain reasonable assumptions [Causal Markov and Causal Faithfulness, see (59)], in a principled way these can be combined into a single coherent causal model. The output consists of a graphical model that depicts the inferred relations between the observed variables, where edges correspond to direct interactions, and arrowheads signify the direction of causal influence. In the current study, we were primarily interested in the direct interaction structure represented by the edges. We chose BCCD because it is accurate, robust, and able to recognize interactions due to unobserved (latent) confounders, while it is able also to test the reliability of individual parts of the output model that will help us assess to what extent the output model follows the HiTOP design. However, any valid causal discovery method that can handle latent variables can be used.

The main principle behind our approach is depicted in Figure 2. When one considers a system with an underlying generating structure like the one depicted in the top left-hand side of Figure 2, with a single overarching top variable (p-factor), that affects a small set of intermediate variables (higher order factors), that each in turn determine a number of leaf variables (the separate RC scales/clusters). If there are no other direct interactions in the system and all variables are observed, then a causal discovery algorithm will be able to fully reconstruct the underlying graph. The same holds true if the relations follow the bifactor model depicted in the bottom left-hand side. Unfortunately, even though this matches our postulated HiTOP structure, we cannot verify this directly because: (1) the true p-factor is not observed directly, and (2) the clustering of RC-scales into higher-order factors is part of our hypothesis.

When a causal discovery algorithm is run on the same system but without p-factor or higher-order factors present in the data, then the many associations that exist between scales in the same factor, as well as between the separate scales from different factors will appear as induced cross links in the overall model. This effect on the output is depicted in the top right of Figure 2. If our hypothesis is correct and the p-factors constructed via the procedure described above is a sufficiently adequate proxy for the truest p-factor, then adding that proxy into the model should resolve and/or significantly weaken many of the induced spurious links between scales from different factors, leading to the output depicted in the bottom right of Figure 2. Therefore, if we do find this model behavior in our causal model output on application to the HiTOP data we can likely take it as a confirmation of the validity of both the global p-factor hypothesis as well as the approximation procedure above.

Finally, if we find good independent confirmation of the HiTOP structure in combination with the p-factor as indicated above, then we can repeat our causal network analysis with the predefined higher-order factors included to infer the “natural” structure of the system as a whole. In particular, we can then expect to find a model that either resembles the traditional hierarchical model in the top left side of Figure 2, or one that is more closely aligned to the so-called bifactor model in the bottom left. Naturally, a preference for either is not yet definitive proof, but it should at least make for a compelling argument of the most likely explanation for the given data, as it is strictly based on identifiable structural properties of the underlying generating system.

As the approach is entirely data driven, observing this behavior in a large, high-quality data set will provide strong independent verification for both the design of the MMPI-2-RF (see Figure 1) as well as the existence and validity of the constructed hypothesized p-factor within the HiTOP model. For the BCCD, regressive scores on the p-factor were used as well as raw scores on all RC scales and H-O scales. We therefore input the full-item raw RC-scales, the two raw 21-item p-factor into the analyses, as well as the full-item raw H-O scales (in certain models). We also entered a construct indicating between clinical and normative samples, as well as age and gender, to identify structural differences in structure, if any. This follows the recently published Joint Causal Inference (JCI) framework for combining data from multiple studies into a single coherent causal model (73).

Results

p-factor scales construction

We identified two sets of 21 items, one for the p-factor with RCd items included (i.e., based on 192 items; hereafter referred to as p-factor_rcd) and one for the p-factor scale without RCd items (i.e., based on 168 items, hereafter referred to as p-factor). For an overview of the originating scales of the highest loading items including factor loadings, we refer to Table 2. The 21 items making up the p-factor_rcd scale were all items originating from the RCd scale, with the exception of one RC7 item. The 21 items making up the p-factor scale were items from 7 different RC scales, not representing RC3. The Chi square tests of model fit indicated significant results for both the p-factor_rcd [χ² (df = 18, 144) = 141,535.73, p < 0.001] and p-factor [χ² (df = 13, 860) = 114,221.44, p < 0.001], which may be due to overpowering.

TABLE 2

Table 2. Originating scales and factor loadings for the p-factor scales based on the MMPI-2 booklet.

Correlational results of the two p-factor scales with the RC scales and PSY-5-r scales are displayed in Table 3. These indicate significant correlations of varying strength in all three samples with nearly all RC-scales and PSY-5-r scales. However, it is important to note this may be inflated due to partial item overlap between the RC scales and the p-factor scales. The exceptions to the overwhelming significant correlations were the scales AGGR-r (measuring the maladaptive personality trait Aggressiveness), DISC-r (measuring the maladaptive personality trait Disconstraint) and INTR-r (measuring the maladaptive trait Introversion/Low Positive Emotionality), which did not significantly correlate with the p-factor scales in all subsamples. Reliability analysis indicated that the p-factor scale with demoralization items had good to excellent in the Combined sample (Cronbach's α = 0.94), Normative sample (Cronbach's α = 0.88), and Clinical sample (Cronbach's α = 0.91). The p-factor scale without demoralization items demonstrated acceptable to good internal consistency in the Combined sample (Cronbach's α = 0.87), Normative sample (Cronbach's α = 0.74), and Clinical sample (Cronbach's α = 0.82).

TABLE 3

Table 3. Correlations with RC, H-O, and PSY-5-r scales and reliability statistics for the p-factor scales by sample and subsample.

BCCD structure analysis

Key BCCD model results are presented in Figures 3–7, which depict results in the combined sample (N = 5,708) only. For clarity reasons, we omitted any inferred causal directions in the output, as we are primarily interested in the emerging data driven clustering patterns. All original output data and figures are available upon request to the first author. Figure 3 displays clustering of the eight RC scales (excluding RCd) without adding either p-factor scale to the model. The results indicate clustering into the spectra as hypothesized, but at the same time show a number of cross-links between scales of different spectra. As a result, clustering patterns emerge, but with a high degree of overlap between scales and/or the presence of latent factors inducing spurious connections. In particular, the connections between RC1 and RC2, between RC7 and RC6, between RC7 and RC2 show this pattern of cross-linking between spectra.

FIGURE 3

Figure 3. Graphical display of the structural organization of the RC scales without p-factor scales. Depicting RC-scales for the Combined Sample (N = 5,708).

Figure 4 displays the results of Figure 3, but with the p-factor scale without demoralization items added to the model. As can be observed from the graphical display, the pattern of clustering is more evident than without p-factor scale added to the model. Several cross-links, including between RC1 and RC2, between RC7 and RC6, and between RC7 and RC2 disappeared as a result. The p-factor scale without demoralization items captures variance between several RC scales to make the clusters more pronounced. The results show a more defined clustering of scales into the hypothesized six spectra (indicated by color): somatoform (purple; RC1), detachment (light blue; RC2), internalizing (dark blue, RC7), thought dysfunction (green: RC6 and RC8), disinhibited-externalizing (red: RC4), and antagonistic-externalizing [orange: RC9 and to a degree RC3 (yellow)]. Importantly, the p-factor scale (without demoralization items) is independently associated with these spectra, as visualized by the orange/yellow edges. Figure 5 illustrates the same results as Figure 4, but with the RCd scale added. As is evident from this model, the RCd scale positions within the internalizing and detachment clusters in the presence of the p-factor. This indicates the RCd scale is most strongly associated with these spectra.

FIGURE 4

Figure 4. Graphical display of the structural organization of eight RC scales with p-factor without RCd items. Depicting RC-scales and p-factor scale without RCd items for the Combined Sample (N = 5,708). Yellow edges indicate p-factor links; black edges indicate cross-links between RC scales.

FIGURE 5

Figure 5. Graphical display of the structural organization of the nine RC scales (including RCd) with p-factor without RCd items. Depicting RC-scales and p-factor scale without RCd items for the Combined Sample (N = 5,708). Yellow edges indicate p-factor links; black edges indicate cross-links between RC scales.

Figure 6 shows the eight RC scales with the p-factor with demoralization items added to the model. Potentially due to this p-factor scale's close resemblance to the RCd scale (20 items out of 21 originate from RCd), the results indicate this p-factor is most strongly associated with the somatoform, internalizing, and detachment spectra. In this model, the RC1–RC8 link does not disappear, nor the association between RC7 and RC8. While the p-factor with RCd items behaves as an overarching p-factor scale to a degree, it mostly captures cross-variance in the internalizing/detachment side of the model.

FIGURE 6

Figure 6. Graphical display of the structural organization of the eight RC scales with p-factor with RCd items. Depicting RC-scales and p-factor scale with RCd items for the Combined Sample (N = 5,708). Yellow edges indicate p-factor links; black edges indicate cross-links between RC scales.

To assess the bifactor vs. hierarchical underlying structures of the general psychopathology theories, Figure 7 shows the RC scales and Higher-Order scales with the p-factor without demoralization items. The results show that the p-factor is not associated with the Higher-Order constructs directly, but is associated with the RC level subfactors. This supports the bifactor model, as in a hierarchical model the p-factor would have directly associated with the Higher-Order scales.

FIGURE 7

Figure 7. Graphical display of the structural organization of the nine RC scales, H-O scales, with p-factor without RCd items. Depicting RC-scales, H-O scales, and p-factor scale without RCd items for the Combined Sample (N = 5,708). Yellow edges indicate p-factor links; black edges indicate cross-links between RC and H-O scales.

Finally, these models were reconstructed for the clinical and normative samples independently, which evidenced only slightly different but robust structures similar to Figures 3–7. This indicates these findings are consistent across samples. In addition, we added age and gender to the model, which did not impact the structure of these models. For efficiency's sake, we have not depicted these models, but all models are available upon request.

Taken together, these results provide support for (1) clustering of subfactors (RC scales) into six spectra (all three samples), (2) the p-factor scale without demoralization items representing a general psychopathology factor due the fewest cross-links between the six spectra, (3) these findings being consistent across samples (normative/clinical) and with or without controlling for age and gender, and (4) the p-factor with demoralization items behaves itself as the RCd scale, which is as expected due to the high item overlap with the scale and one item from RC7 (internalizing spectrum).

Discussion

We investigated the hierarchical structure of the MMPI-2-RF scales, using an underutilized Bayesian explorative approach. First, we extracted two scales as proxies of general psychopathology (p-factors), one with and one without demoralization items in the scale. This resulted in one scale with many demoralization items and one with items from 7 different RC scales represented. Next, we conducted a bottom-up constructed model, which resulted in six clusters (i.e., internalizing, disinhibited-externalizing, antagonistic-externalizing, thought disorder, detachment, and somatoform; see Figures 4, 5 in particular) mapping onto the spectra in the HiTOP-model (3, 14). The bottom-up clustering of the RC scale scores into spectra demonstrated a more pronounced structure after including the p-factor without demoralization items. These models are unique, as they are entirely data-driven and not dependent on the molding process of traditional factor analytic approaches (74, 75). We used a novel statistical approach in two large samples of both community and clinical origin.

Additionally, the results of the present study fit with the bifactor model as described in the literature (54, 55). As seen in comparing Figures 3, 4 as well as in Figure 7, the cross-links between RC-scales become less pronounced and connections between the spectra persist. This effect is evident when comparing models with and without p-factor: the spectra explain common variance between the RC-dimensions, indicating this to be a valid hierarchical level when constructing the dimensional structure bottom-up (54). Overall, our findings add to validity research of the MMPI-2-RF as a tool to hierarchically and multidimensionally operationalize the HiTOP.

Hierarchical organization of the super spectrum

As hypothesized, the super spectrum without demoralization items was found to be an all-encompassing construct relative to spectrum-constructs in the Bayesian model, which emphasized the overarching nature of this construct. We constructed this p-factor scale without demoralization items, whereas the p-factor with demoralization items mostly resembled the RCd scale itself, despite inclusion of all RC items in the scale construction process. A single latent construct captured significant common variance among the symptomatology over and beyond the higher order factors, in line with earlier findings on the p-factor (1, 3). Theoretically it is of interest to investigate the “comorbid” patterns among individuals suffering from different forms of psychopathology, which appears to be demonstrated in at least six phenotypes of psychopathology according to these results (i.e., internalizing, disinhibited-externalizing, antagonistic-externalizing, thought disorder, detachment, and somatoform). Due to these spectrum-super spectrum relationships, the p-factor constructs should be seen in context of the larger model of hierarchical psychopathology dimensions, rather than as an autonomously meaningful construct (3, 53, 76). We found associations between the p-factor without demoralization construct and demoralization as a construct (Figure 5), despite the absence of item-overlap. However, when conducting these analyses with the p-factor with Demoralization items, the p-factor construct did not behave as an as overarching construct, which the p-factor without Demoralization items did. This may indicate the Demoralization construct is mostly associated with somatoform, internalizing (neuroticism), and detachment spectra, as was found in earlier research (19).

On an item-level, the p-factor without demoralization items seems to entail varying kinds of psychopathology, including emotional dysregulation, paranoid symptoms, various externalizing, and hypomanic symptoms (see Tables 2, 3). In the models with the p-factor without demoralization items, this scale functions as a broker-construct of the different spectra. Importantly, our findings indicate the model structure is much clearer in presence of a general p-factor without demoralization, which is in line with several previous studies (1, 77). We identified a clear tendency toward the bifactor model of psychopathology dimensions, rather than the standard hierarchical model (55), because second tier factors (higher-order factors) do not mediate between the p-factor and RC-scales, and appear disjoint from the p-factor when included in the full model.

Hierarchical organization of the six spectra

The results in the present study replicate the hierarchical structure of dimensional measures of psychopathology, in line with the bifactor model (54). Importantly, in presence of a general psychopathology factor or super spectrum without demoralization items, 8 RC scales representing dimensional measures of psychopathology clustered into six spectra (21, 44, 78). These spectra map onto the six spectra described in the HiTOP model and are in line with previous research (14, 19).

First, Dysfunctional Negative Emotions (RC7) and—in some models—Demoralization (RCd), formed an internalizing spectrum, which is in line with previous factor analytic studies (20, 44). The Low Positive Emotions construct (RC2) could be seen as the detachment spectrum on its own since it did not associate with RC7 but directly with the p-factor (Figure 6), as well as with Hypomanic Activation (RC9). When adding the Demoralization construct, it did cluster into a more pronounced internalizing spectrum in the p-factor model, which may underline the demoralization component of the internalizing spectrum specifically. The Demoralization construct in this instance mediated between the internalizing spectrum (i.e., RC7) and the p-factor. Second, Antisocial Behavior (RC4) formed into a disinhibited-externalizing spectrum, whereas the Hypomanic Activation (RC9) scale and to a degree the Cynicism (RC3) scale formed an antagonistic-externalizing cluster. The two externalizing spectra were closely related; however, which replicates previous results indicating that instrumental acting-out and narcissistic, antagonistic types of externalizing symptomatology load on the same latent construct (70, 79). Further, Aberrant Experiences (RC8) and Ideas of Persecution (RC6) [and to a degree Cynicism (RC3)] clustered into a thought dysfunction spectrum, as is expected and in line with the literature (13). Finally, though somewhat associated to the internalizing spectrum, we found a somatoform spectrum, formed by Somatic Complaints [RC1, cf. (55)].

Taken together, these findings suggest a hierarchical factor structure on the spectrum level, not excluding the possibility of other spectra not measured in the present study, such as an obsessiveness subfactor or maladaptive trait (Anankastia). The utilization of BCCD as a probabilistic alternative to traditional factor-analytic approaches has allowed for independent validation of the three-factor model (33, 44), with the addition of a distinct somatoform spectrum, detachment spectrum, and splitting of the externalizing spectrum into antagonistic and disinhibited. With the publication of the new MMPI-3 (12), higher level constructs could be utilized as meaningful measures of psychopathology, potentially expanding on the already existing three higher-order scales of the MMPI-2-RF.

We discovered several anomalous yet interesting characteristics in the models. Cynicism (RC3) was diffusely positioned in relation to the spectra with overlapping associations with internalizing, thought dysfunction, and antagonistic-externalizing spectra. Cynicism as a construct does not refer to psychopathology in a narrow sense but encompasses other-referential beliefs. Despite clustering into the internalizing and thought dysfunction spectra respectively, Dysfunctional Negative Emotions (RC7) and Cynicism (RC3) were found to be associated in most models. This connection is presumed to be the result of conceptual overlap, or the co-occurrence of constructs measured by both scales that might be labeled as indignation or resentfulness, which captures both negative emotionality and paranoid tendencies (i.e., the feeling that one is treated unfairly) as both scales demonstrate moderate to strong relations with paranoid personality features (39, 70, 71). The latter may be due to the notion that the construct Cynicism (as captured in the MMPI-2-RF) seems to be a more diffuse amalgamation of overlapping aspects between the higher order factors, and persists as an extremely stable pattern, with or without the p-factor without demoralization items. Further, we found a strong relationship between Cynicism (RC3) and Hypomanic Activation (RC9), potentially resulting from common features regarding negative attitudes toward others, either due to narcissistic features or distrustful beliefs; and possibly underlying is self-referential mistrust (17). This is expressed in the antagonistic-externalizing spectrum. It is important to note that Cynicism (RC3) was not represented in the p-factor scale.

A robust association emerged between Low Positive Emotionality (RC2) and Hypomanic Activation (RC9). We suspect this to be a product of clinical overlap, including social disengagement and lack of energy—symptoms of Low Positive Emotionality (RC2), linked to introversion—and overactivity and restlessness due to hypomanic symptomatology, linked to extraversion (80). Moreover, Detachment can be formed by Low Positive Emotionality (RC2) by itself, which further explains this association (14). Low Positive Emotionality (RC2) is also associated with Aberrant Experiences (RC8), possibly due to psychotic depressive symptoms and negative symptoms (avolition) in the clinical sample. We found a weaker association between Ideas of Persecution (RC6) and Hypomanic Activation (RC9), both characteristics of bipolar disorder with psychotic features. In addition, paranoid ideas co-occur with narcissistic and antagonistic personality features (81). The aforementioned cross-spectrum relationships underline the notion that the spectra are not isolated dimensions, but rather that there are commonalities between key psychopathology dimensions on lower levels of the hierarchy (76).

Clinical implications of the present findings

Our findings have several clinical implications. First, the outcomes support validity of the MMPI-2-RF scales, as well as the MMPI-2-RF's hierarchical structure of these measures. Second, latent constructs on a spectrum or super spectrum level were found in a combined clinical and normative sample, underlining the robustness of this hierarchical dimensional model of psychopathology. These dimensions could be targeted by spectrum-wide interventions. For example, the Unified Protocol for Emotional Disorders (82) may target the internalizing and detachment spectra, such as anxiety and mood symptomatology (i.e., “emotional disorders”). Targeting the common spectra in transdiagnostic treatment rather than having different sequential treatments targeting different forms of symptomatology seems not only be equally effective clinically [e.g., (83)], but also more cost-efficient for both the individual and for the health care system at large.

Second, as suggested by Caspi et al. (1) an elevated p-factor scale score may indicate a risk for several types of psychopathology for both the individual, but also for their family members. The relationship with Demoralization in our results may imply that high p-factor levels could indicate a profound mental illness risk, family members who are at risk, and lower quality of life as a result (1). Clinicians could keep these implications in mind if this construct were to be incorporated into diagnostic paradigms or manuals. Adequate measurement of a more developed p-factor construct may possibly be a meaningful addition to diagnostic assessment, particularly in light of transdiagnostic treatments targeting psychopathology overall (84). As opposed to the top-down development of general intelligence g and cognitive domains consecutively, the present study emphasizes the “bottom up” construction of a general psychopathology factor from underlying spectra. However, we urge clinicians not to reify the p-factor and not to view a potential general psychopathology scale as a stand-alone clinical tool, let alone the scales developed in the present study. As argued by several scholars [e.g., (3, 76)], the super spectrum is a part of a larger taxonomy of dimensions and should therefore not be regarded as clinically all-encompassing as such a single score may have a similar stigmatizing effect to multiple categorical labels.

Limitations and future directions

The present study has several constraints that warrant noting. Regarding our statistical procedures, our main limitation is overlap between the items of the RC scales and the constructed p-factor scales. This may overestimate associations between the p-factor scales and different clinical constructs. Further, we were not able to cross-validate our p-factor measures with external measures, limiting the validity of these scales. The p-factor scales constructed were mere proxies and produced only with exploratory factor analysis. Additionally, we solely relied on the RC scales as measure of psychopathology, which—despite its well-established validity and reliability—may limit the generalizability of results. For example, compulsivity (i.e., the Anankastic domain) is not measured by the RC scales, despite this being a clinically relevant spectrum among other personality disorder domains described in the ICD-11 (85). Further research using all hierarchical levels, including the SP-scales, new MMPI-3 scales, and PSY-5-r scales, may elucidate the structure even more comprehensively [e.g., (13)]. Eating disorder symptomatology as well as sexual dysfunctions and disorders are not directly measured by the MMPI-2-RF, but may possible form other distinct spectra. Also, the MMPI is based on self-report to measure pathology, which can be seen as indirect and potentially biased.

We encourage future scientific efforts to be focused on further replication of the HiTOP model and dimensional measurement of psychopathology using novel methods, with a specific focus on bifactor vs. hierarchical nature of these models. While this wasn't the goal of the present study, more research would be needed to construct and validate a psychometrically sound p-factor scale and dimension, particularly given the finding that multicollinear scales will likely share a latent factor (53, 86). The benefits of the independent analytic approach based on causal model discovery could be brought to other applications or areas of research in psychopathology and beyond. Not because it is fundamentally “superior” than our existing tools, but because it can provide an alternative approach to available data that may offer new insights in the relevant underlying processes. The latter can be difficult to obtain in any other way. Alternatively, it could be an effective tool for independent confirmation of existing models. BCCD could play a role as an initial exploratory means of analysis for new data sets, as well as help formulate new hypotheses by virtue of an intuitive graphical representation of a system.

Conclusions

Using a highly innovative, inferential statistical method, we produced an entirely data driven model in which constructs of psychopathology clustered into six spectra, replicating the HiTOP's main spectra in a combined normative and clinical sample. Moreover, p-factor proxies were constructed as super spectra (3), resulting in the finding that one p-factor (not representing Cynicism and Demoralization) may be underlying to six syndromic constructs of psychopathology.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: the datasets are not publicly available. The corresponding author can be contacted for more information. Requests to access these datasets should be directed to RL, robbert.langwerden@donders.ru.nl.

Ethics statement

The studies involving human participants were reviewed and approved by Institutional Review Board of Reinier van Arkel Mental Health Institute. The patients/participants provided their written informed consent to participate in this study.

Author contributions

RL, PH, TC, and JE contributed to the study conception and design. Material preparation and data collection was performed by PH, JE, and JD. Data analyses were performed by RL, PH, and TC and were verified by the rest of the authors. The first draft of the manuscript was written by RL and TC. All authors commented on various versions of the manuscript. All authors read and approved the final manuscript.

Acknowledgments

The authors wish to extend their gratitude to Huub Schaeks and Theo Bögels for their support in this research project. RL acknowledges he holds the position of research analyst at Florida International University in addition to his external graduate student position at the Donders Institute of Brain, Cognition, and Behaviour at the Radboud University Nijmegen.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Caspi A, Houts RM, Belsky DW, Goldman-Mellor SJ, Harrington H, Israel S, et al. The p factor: one general psychopathology factor in the structure of psychiatric disorders? Clin Psychol Sci. (2014) 2:119–37. doi: 10.1177/2167702613497473

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Haslam N, Holland E, Kuppens P. Categories versus dimensions in personality and psychopathology: a quantitative review of taxometric research. Psychol Med. (2012) 42:903–20. doi: 10.1017/S0033291711001966

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Kotov R, Krueger RF, Watson D, Achenbach TM, Althoff RR, Bagby RM, et al. The Hierarchical Taxonomy of Psychopathology (HiTOP): a dimensional alternative to traditional nosologies. J Abnorm Psychol. (2017) 126:454. doi: 10.1037/abn0000258

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Hopwood CJ, Kotov R, Krueger RF, Watson D, Widiger TA, Althoff RR, et al. The time has come for dimensional personality disorder diagnosis. Pers Ment Health. (2018) 12:82–6. doi: 10.1002/pmh.1408

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Krueger RF, Markon KE. The role of the DSM-5 personality trait model in moving toward a quantitative and empirically based approach to classifying personality and psychopathology. Annu Rev Clin Psychol. (2014) 10:477–501. doi: 10.1146/annurev-clinpsy-032813-153732

PubMed Abstract | CrossRef Full Text | Google Scholar

6. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5^®), 5th edition. Washington, DC: American Psychiatric Hub (2013). doi: 10.1176/appi.books.9780890425596

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Conway CC, Forbes MK, Forbush KT, Fried EI, Hallquist MN, Kotov R, et al. A hierarchical taxonomy of psychopathology can transform mental health research. Perspect Psychol Sci. (2019) 14:419–36. doi: 10.1177/1745691618810696

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Haeffel GJ, Jeronimus BF, Kaiser BN, Weaver LJ, Soyster PD, Fisher AJ, et al. Folk classification and factor rotations: whales, sharks, and the problems with the Hierarchical Taxonomy of Psychopathology (HiTOP). Clin Psychol Sci. (2021) 10:259–78. doi: 10.1177/21677026211002500

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Krueger RF, Kotov R, Watson D, Forbes MK, Eaton NR, Ruggero CJ, et al. Progress in achieving quantitative classification of psychopathology. World Psychiatry. (2018) 17:282–93. doi: 10.1002/wps.20566

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Ruggero CJ, Kotov R, Hopwood CJ, First M, Clark LA, Skodol AE, et al. Integrating the hierarchical taxonomy of psychopathology (HiTOP) into clinical practice. J Consult Clin Psychol. (2019) 87:1069–84. doi: 10.1037/ccp0000452

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Ben-Porath YS, Tellegen A. Minnesota Multiphasic Personality Inventory-2 Restructured Form. Minneapolis, MN: University of Minnesota Press (2008). doi: 10.1037/t15121-000

CrossRef Full Text | Google Scholar

12. Ben-Porath YS, Tellegen A. Minnesota Multiphasic Personality Inventory-3 (MMPI-3): Technical Manual. Minneapolis, MN: University of Minnesota Press (2020).

Google Scholar

13. Sellbom M, Kremyar AJ, Wygant DB. Mapping MMPI-3 scales onto the Hierarchical Taxonomy of Psychopathology. Psychol Assess. (2021) 33:1153–68. doi: 10.1037/pas0001049

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Sellbom M. The MMPI-2-Restructured Form (MMPI-2-RF): assessment of personality and psychopathology in the twenty-first century. Annu Rev Clin Psychol. (2019) 15:149–77. doi: 10.1146/annurev-clinpsy-050718-095701

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Tellegen A, Ben-Porath YS, McNulty J, Arbisi P, Graham J, Kaemmer B. MMPI-2 Restructured Clinical (RC) Scales: Development, Validation, and Interpretation. Minneapolis, MN: University of Minnesota Press (2003).

PubMed Abstract | Google Scholar

16. Harkness AR, McNulty JL, Finn JA, Reynolds SM, Shields SM, Arbisi P. The MMPI-2-RF Personality Psychopathology Five (PSY-5-RF) scales: development and validity research. J Pers Assess. (2014) 96:140–50. doi: 10.1080/00223891.2013.823439

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Sellbom M, Ben-Porath YS, Bagby RM. Personality and psychopathology: mapping the MMPI-2 restructured clinical (RC) scales onto the five factor model of personality. J Pers Disord. (2008) 22:291–312. doi: 10.1521/pedi.2008.22.3.291

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Stanley IH, Yancey JR, Patrick CJ, Joiner TE. A distinct configuration of MMPI-2-RF scales RCd and RC9/ACT is associated with suicide attempt risk among suicide ideators in a psychiatric outpatient sample. Psychol Assess. (2018) 30:1249. doi: 10.1037/pas0000588

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Arbisi PA, Sellbom M, Ben-Porath YS. Empirical correlates of the MMPI-2 restructured clinical (RC) scales in psychiatric inpatients. J Pers Assess. (2008) 90:122–8. doi: 10.1080/00223890701845146

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Brown TA, Barlow DH. A proposal for a dimensional classification system based on the shared features of the DSM-IV anxiety and mood disorders: implications for assessment and treatment. Psychol Assess. (2009) 21:256. doi: 10.1037/a0016608

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Kotov R. New dimensions in the quantitative classification of mental illness. Arch Gen Psychiatry. (2011) 68:1003. doi: 10.1001/archgenpsychiatry.2011.107

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Keyes KM, Eaton NR, Krueger RF, Skodol AE, Wall MM, Grant B, et al. Thought disorder in the meta-structure of psychopathology. Psychol Med. (2013) 43:1673–83. doi: 10.1017/S0033291712002292

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Markon KE. Modeling psychopathology structure: a symptom-level analysis of Axis I and II disorders. Psychol Med. (2010) 40:273–88. doi: 10.1017/S0033291709990183

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Eaton NR. Measurement and mental health disparities: psychopathology classification and identity assessment. Pers Ment Health. (2020) 14:76–87. doi: 10.1002/pmh.1449

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Shankman SA, Funkhouser CJ, Klein DN, Davila J, Lerner D, Hee D. Reliability and validity of severity dimensions of psychopathology assessed using the Structured Clinical Interview for DSM-5 (SCID). Int J Methods Psychiatr Res. (2018) 27:e1590. doi: 10.1002/mpr.1590

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Markon KE, Chmielewski M, Miller CJ. The reliability and validity of discrete and continuous measures of psychopathology: a quantitative review. Psychol Bull. (2011) 137:856. doi: 10.1037/a0023678

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Luyten P, Fonagy P. Integrating and differentiating personality and psychopathology: a psychodynamic perspective. J Pers. (2022) 90:75–88. doi: 10.1111/jopy.12656

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Tyrer P. Comorbidity, consanguinity and co-occurrence. BJPsych Adv. (2017) 23:167–8. doi: 10.1192/apt.bp.116.016444

CrossRef Full Text | Google Scholar

29. Krueger RF, Piasecki TM. Toward a dimensional and psychometrically-informed approach to conceptualizing psychopathology. Behav Res Ther. (2002) 40:485–99. doi: 10.1016/S0005-7967(02)00016-5

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Kotov R, Gamez W, Schmidt F, Watson D. Linking “big” personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. Psychol Bull. (2010) 136:768. doi: 10.1037/a0020327

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Widiger TA, Sellbom M, Chmielewski M, Clark LA, DeYoung CG, Kotov R, et al. Personality in a hierarchical model of psychopathology. Clin Psychol Sci. (2019) 7:77–92. doi: 10.1177/2167702618797105

CrossRef Full Text | Google Scholar

32. DeYoung CG, Chmielewski M, Clark LA, Condon DM, Kotov R, Krueger RF, et al. The distinction between symptoms and traits in the Hierarchical Taxonomy of Psychopathology (HiTOP). J Pers. (2020) 90:20–33. doi: 10.1111/jopy.12593

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Díaz-Batanero C, Aluja A, Sayans-Jiménez P, Baillés E, Fernández-Calderón F, Peri JM, et al. Alternative DSM-5 model for personality disorders through the lens of an empirical network model. Assessment. (2021) 28:773–87. doi: 10.1177/1073191119897118

PubMed Abstract | CrossRef Full Text | Google Scholar

34. McNally RJ. The network takeover reaches psychopathology. Behav Brain Sci. (2019) 42:e15. doi: 10.1017/S0140525X18001073

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Robinaugh DJ, Hoekstra RHA, Toner ER, Borsboom D. The network approach to psychopathology: a review of the literature 2008-2018 and an agenda for future research. Psychol Med. (2020) 50:353–66. doi: 10.1017/S0033291719003404

PubMed Abstract | CrossRef Full Text | Google Scholar

36. DeYoung CG, Kotov R, Krueger RF, Cicero DC, Conway CC, Eaton NR, et al. Answering questions about the hierarchical taxonomy of psychopathology (HiTOP): analogies to whales and sharks miss the boat. Clin Psychol Sci. (2021) 10:279–84. doi: 10.1177/21677026211049390

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Bagby RM, Sellbom M, Ayearst LE, Chmielewski MS, Anderson JL, Quilty LC. Exploring the hierarchical structure of the MMPI-2-RF personality psychopathology five in psychiatric patient and university student samples. J Pers Assess. (2014) 96:166–72. doi: 10.1080/00223891.2013.825623

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Romero IE, Toorabally N, Burchett D, Tarescavage AM, Glassmire DM. Mapping the MMPI-2-RF substantive scales onto internalizing, externalizing, and thought dysfunction dimensions in a forensic inpatient setting. J Pers Assess. (2017) 99:351–62. doi: 10.1080/00223891.2016.1223681

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Van der Heijden PT, Rossi GPM, Van der Veld WM, Derksen JJL, Egger JIM. Personality and psychopathology: higher order relations between the Five Factor Model of Personality and the MMPI-2 Restructured Form. J Res Pers. (2013) 47:572–9. doi: 10.1016/j.jrp.2013.05.001

CrossRef Full Text | Google Scholar

40. Waszczuk MA, Kotov R, Ruggero C, Gamez W, Watson D. Hierarchical structure of emotional disorders: from individual symptoms to the spectrum. J Abnorm Psychol. (2017) 126:613. doi: 10.1037/abn0000264

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Griffith JW, Zinbarg RE, Craske MG, Mineka S, Rose RD, Waters AM, et al. Neuroticism as a common dimension in the internalizing disorders. Psychol Med. (2010) 40:1125–36. doi: 10.1017/S0033291709991449

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Anderson JLS, Ayearst M, Quilty LC, Chmielewski M, Bagby RM. Associations between DSM-5 section III personality traits and the Minnesota Multiphasic Personality Inventory 2-Restructured Form (MMPI-2-RF) scales in a psychiatric patient sample. Psychol Assessment. (2015) 27:801–915. doi: 10.1037/pas0000096

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Kessler RC, Petukhova M, Zaslavsky AM. The role of latent internalizing and externalizing predispositions in accounting for the development of comorbidity among common mental disorders. Curr Opin Psychiatry. (2011) 24:307–12. doi: 10.1097/YCO.0b013e3283477b22

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Vollebergh WAM, Iedema J, Bijl RV, De Graaf R, Smit F, Ormel J. The structure and stability of common mental disorders. Arch Gen Psychiatry. (2001) 58:597. doi: 10.1001/archpsyc.58.6.597

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Blanco C, Wall MM, He JP, Krueger RF, Olfson M, Jin CJ, et al. The space of common psychiatric disorders in adolescents: comorbidity structure and individual latent liabilities. J Am Acad Child Adolesc Psychiatry. (2015) 54:45–52. doi: 10.1016/j.jaac.2014.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Eaton NR, Keyes KM, Krueger RF, Balsis S, Skodol AE, Markon KE, et al. An invariant dimensional liability model of gender differences in mental disorder prevalence: evidence from a national sample. J Abnorm Psychol. (2012) 121:282–8. doi: 10.1037/a0024780

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Krueger RF, Markon KE. Reinterpreting comorbidity: a model-based approach to understanding and classifying psychopathology. Annu Rev Clin Psychol. (2006) 2:111–33. doi: 10.1146/annurev.clinpsy.2.022305.095213

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Selzam S, Coleman JRI, Caspi A, Moffitt TE, Plomin R. A polygenic p factor for major psychiatric disorders. Transl Psychiatry. (2018) 8:1–9. doi: 10.1038/s41398-018-0217-4

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Spearman C. “General Intelligence” Objectively determined and measured. In: Jenkins JJ, Paterson, DG, editors. Studies in Individual Differences: The Search for intelligence. New York, NY: Appleton-Century-Crofts (1961). p. 59–73. doi: 10.1037/11491-006

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Caspi A, Moffitt TE. All for one and one for all: mental disorders in one dimension. Am J Psychiatry. (2018) 175:831–44. doi: 10.1176/appi.ajp.2018.17121383

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Allegrini AG, Cheesman R, Rimfeld K, Selzam S, Pingault JB, Eley TC, et al. The p factor: genetic analyses support a general dimension of psychopathology in childhood and adolescence. J Child Psychol Psychiatry. (2020) 61:30–9. doi: 10.1111/jcpp.13113

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Rosenström T, Gjerde LC, Krueger RF, Aggen SH, Czajkowski NO, Gillespie NA, et al. Joint factorial structure of psychopathology and personality. Psychol Med. (2019) 49:2158–67. doi: 10.1017/S0033291718002982

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Watts AL, Lane SP, Bonifay W, Steinley D, Meyer FAC. Building theories on top of, and not independent of, statistical models: the case of the p-factor. Psychol Inq. (2020) 31:310–20. doi: 10.1080/1047840X.2020.1853476

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Smith GT, Atkinson EA, Davis HA, Riley EN, Oltmanns JR. The general factor of psychopathology. Annu Rev Clin Psychol. (2020) 16:75–98. doi: 10.1146/annurev-clinpsy-071119-115848

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Markon KE. Bifactor and hierarchical models: specification, inference, and interpretation. Annu Rev Clin Psychol. (2019) 15:51–69. doi: 10.1146/annurev-clinpsy-050718-095522

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Southward MW, Cheavens JS, Coccaro EF. Defining the p-factor: an empirical test of five leading theories. Psychol Med. (2022) 2022:1–12. doi: 10.1017/S0033291722001635

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Cumming G. The new statistics: why and how. Psychol Sci. (2014) 25:7–29. doi: 10.1177/0956797613504966

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Claassen T, Heskes T. A Bayesian approach to constraint based causal inference. arXiv pre-print server (2012).

Google Scholar

59. Spirtes P, Glymour CN, Scheines R, Heckerman D. Causation, Prediction, and Search. Cambridge, MA: MIT press (2000).

Google Scholar

60. Hyttinen A, Eberhardt F, Järvisalo M. Constraint-based causal discovery: conflict resolution with answer set programming. UAI. 2014:340–9.

Google Scholar

61. Wechsler D. Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV). San Antonio, TX: NCS Pearson (2008). p. 816–27. doi: 10.1037/t15169-000

CrossRef Full Text | Google Scholar

62. Van Dijk JAM, Claassen T, Suwartono C, Van der Veld WM, Van der Heijden PT, Hendriks MPH. Evaluating WAIS-IV structure through a different psychometric lens: structural causal model discovery as an alternative to confirmatory factor analysis. Clin Neuropsychol. (2017) 31:1141–54. doi: 10.1080/13854046.2017.1352029

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Laceulle OM, Vollebergh WA, Ormel J. The structure of psychopathology in adolescence: Replication of a general psychopathology factor in the TRAILS study. Clin Psychol Sci. (2015) 3:850–60. doi: 10.1177/2167702614560750

CrossRef Full Text | Google Scholar

64. Sellbom M. Mapping the MMPI-2-RF Specific Problems scales onto extant psychopathology structures. J Pers Assess. (2017) 99:341–50. doi: 10.1080/00223891.2016.1206909

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Van der Heijden PT, Egger JIM, Derksen JJL. Comparability of scores on the MMPI-2-RF scales generated with the MMPI-2 and MMPI-2-RF booklets. J Pers Assess. (2010) 92:254–9. doi: 10.1080/00223891003670208

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Derksen JJL, De Mey HRA, Sloore H, Hellenbosch G. MMPI-2: Handleiding bij afname, scoring en interpretative [Minnesota Multiphasic Personality Inventory-2: MMPI-2: Manual for administration, scoring, and interpretation]. Bemmel, Netherlands: PEN Test Publishers (1993).

Google Scholar

67. Van der Heijden PT, Egger JIM, Rossi GPM, Grundel G, Derksen JJL. The MMPI-2-Restructured Form and the standard MMPI-2 clinical scales in relation to DSM-IV. Eur J Psychol Assess. (2013) 29:182–8. doi: 10.1027/1015-5759/a000140

CrossRef Full Text | Google Scholar

68. Van der Heijden PT, Derksen JJL, Egger JIM, Rossi GPM, Laheij M, Bögels T. MMPI-2-RF. Minnesota Multiphasic Personality Inventory-2 Restructured Form Handleiding voor Afname, scoring en interpretatie [Manual for administration, scoring and interpretation: Dutch-Flemish translation]. Bemmel, Netherlands: PEN Tests Publisher. (2013).

Google Scholar

69. Van der Heijden PT, Egger JIM, Derksen JJL. Psychometric evaluation of the MMPI-2 Restructured Clinical scales in two Dutch samples. J Pers Assess. (2008) 90:456–64. doi: 10.1080/00223890802248745

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Van der Heijden PT, Egger JIM, Rossi GPM, Derksen JJL. Integrating psychopathology and personality disorders conceptualized by the MMPI-2-RF and the MCMI-III: a structural validity study. J Pers Assess. (2012) 94:345–57. doi: 10.1080/00223891.2012.656861

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Van der Heijden PT, Egger JIM, Rossi GMP, Van der Veld WM, Derksen JJL. Personality and psychopathology: mapping the MMPI-2-RF on Cloninger's Psychobiological Model of Personality. Assessment. (2013) 20:576–84. doi: 10.1177/1073191113490791

PubMed Abstract | CrossRef Full Text | Google Scholar

72. IBM Corp. IBM SPSS Statistics for Windows, Version 28.0. Armonk, NY: IBM Corp (2021).

Google Scholar

73. Mooij JM, Magliacane S, Claassen T. Joint causal inference from multiple contexts. J Mach Learn Res. (2020) 21:1–108. doi: 10.48550/arXiv.1611.10351

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Borsboom D, Cramer AOJ. Network analysis: an integrative approach to the structure of psychopathology. Annu Rev Clin Psychol. (2013) 9:91–121. doi: 10.1146/annurev-clinpsy-050212-185608

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Borsboom D. A network theory of mental disorders. World Psychiatry. (2017) 16:5–13. doi: 10.1002/wps.20375

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Wright AG, Simms LJ. On the structure of personality disorder traits: conjoint analyses of the CAT-PD, PID-5, and NEO-PI-3 trait models. Personal Disord. (2014) 5:43–54. doi: 10.1037/per0000037

PubMed Abstract | CrossRef Full Text | Google Scholar

77. Lahey BB, Krueger RF, Rathouz PJ, Waldman ID, Zald DH. A hierarchical causal taxonomy of psychopathology across the life span. Psychol Bull. (2017) 143:142. doi: 10.1037/bul0000069

PubMed Abstract | CrossRef Full Text | Google Scholar

78. Marek RJ, Anderson JL, Tarescavage AM, Martin-Fernandez K, Haugh S, Block AR, Ben-Porath YS. Elucidating somatization in a dimensional model of psychopathology across medical settings. J Abnorm Psych. (2020) 129:162. doi: 10.1037/abn0000475

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Witkiewitz K, King K, McMahon RJ, Wu J, Luk J, Bierman KL, et al. Evidence for a multi-dimensional latent structural model of externalizing disorders. J Abnorm Child Psychol. (2013) 41:223–37. doi: 10.1007/s10802-012-9674-z

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Goldberg DP, Krueger RF, Andrews G, Hobbs MJ. Emotional disorders: cluster 4 of the proposed meta-structure for DSM-V and ICD-11: Paper 5 of 7 of the thematic section: ‘a proposal for a meta-structure for DSM-V and ICD-11'. Psychol Med. (2009) 39:2043–59. doi: 10.1017/S0033291709990298

PubMed Abstract | CrossRef Full Text | Google Scholar

81. Millon T, Davis RO. Disorders of Personality: DSM-IV and Beyond. Hoboken, NJ: John Wiley & Sons (1996).

Google Scholar

82. Farchione TJ, Fairholme CP, Ellard KK, Boisseau CL, Thompson-Hollands J, Carl JR, et al. Unified Protocol for transdiagnostic treatment of emotional disorders: a randomized controlled trial. Behav Ther. (2012) 43:666–78. doi: 10.1016/j.beth.2012.01.001

PubMed Abstract | CrossRef Full Text | Google Scholar

83. Steele SJ, Farchione TJ, Cassiello-Robbins C, Ametaj A, Sbi S, Sauer-Zavala S, et al. Efficacy of the Unified Protocol for transdiagnostic treatment of comorbid psychopathology accompanying emotional disorders compared to treatments targeting single disorders. J Psychiatr Res. (2018) 104:211–6. doi: 10.1016/j.jpsychires.2018.08.005

PubMed Abstract | CrossRef Full Text | Google Scholar

84. Sakiris N, Berle D. A systematic review and meta-analysis of the Unified Protocol as a transdiagnostic emotion regulation based intervention. Clin Psychol Rev. (2019) 72:101751. doi: 10.1016/j.cpr.2019.101751

PubMed Abstract | CrossRef Full Text | Google Scholar

85. Bach B, Sellbom M, Kongerslev M, Simonsen E, Krueger RF, Mulder R. Deriving ICD-11 personality disorder domains from DSM-5 traits: initial attempt to harmonize two diagnostic systems. Acta Psychiatr Scand. (2017) 136:108–17. doi: 10.1111/acps.12748

PubMed Abstract | CrossRef Full Text | Google Scholar

86. Van Bork R, Epskamp S, Rhemtulla M, Borsboom D, Van der Maas HLJ. What is the p-factor of psychopathology? Some risks of general factor modeling. Theory Psychol. (2017) 27:759–73. doi: 10.1177/0959354317737185

PubMed Abstract | CrossRef Full Text | Google Scholar