Examining the Dynamic Structure of Daily Internalizing and Externalizing Behavior at Multiple Levels of Analysis

Psychiatric diagnostic covariation suggests that the underlying structure of psychopathology is not one of circumscribed disorders. Quantitative modeling of individual differences in diagnostic patterns has uncovered several broad domains of mental disorder liability, of which the Internalizing and Externalizing spectra have garnered the greatest support. These dimensions have generally been estimated from lifetime or past-year comorbidity patters, which are distal from the covariation of symptoms and maladaptive behavior that ebb and flow in daily life. In this study, structural models are applied to daily diary data (Median = 94 days) of maladaptive behaviors collected from a sample (N = 101) of individuals diagnosed with personality disorders (PDs). Using multilevel and unified structural equation modeling, between-person, within-person, and person-specific structures were estimated from 16 behaviors that are encompassed by the Internalizing and Externalizing spectra. At the between-person level (i.e., individual differences in average endorsement across days) we found support for a two-factor Internalizing–Externalizing model, which exhibits significant associations with corresponding diagnostic spectra. At the within-person level (i.e., dynamic covariation among daily behavior pooled across individuals) we found support for a more differentiated, four-factor, Negative Affect-Detachment-Hostility-Disinhibition structure. Finally, we demonstrate that the person-specific structures of associations between these four domains are highly idiosyncratic.

Introduction

Occasioned by patterns of extensive diagnostic co-occurrence, there has been substantial interest in mapping the fundamental nature of psychopathology using quantitative modeling techniques (e.g., Krueger, 1999; Krueger and Markon, 2006; Cramer et al., 2010; Borsboom et al., 2011; Kotov et al., 2011; Wigman et al., 2015). Prime examples of these efforts include the empirically identified Internalizing (e.g., unipolar mood disorders, anxiety disorders) and Externalizing (e.g., substance use, antisocial behavior) spectra (e.g., Achenbach, 1966; Krueger, 1999; Wright et al., 2013). As has been the case in the basic personality trait literature, research on the structure of mental disorders has prioritized the between-person level of analysis (i.e., individual differences). However, there has been increasing interest in studying contextualized dynamic processes associated with psychopathology (e.g., Myin-Germeys et al., 2009; Wichers, 2014). These approaches use a variety of within-person data collection and analytic techniques that seek to illuminate the granular and nuanced dynamics of mental disorders. On the surface these two perspectives to understanding psychopathology may seem at odds: one seeking to cast clinical phenomena in terms of generalities, the other pursuing a high degree of specificity. Here we explore bridging these two approaches by examining the structures that emerge from daily diary reports of maladaptive behaviors at the between-person, within-person, and person-specific levels of analysis. In so doing we draw links to efforts in basic personality science that seek to integrate structural and dynamic models by treating traits as ensembles of contextualized processes (e.g., Wright, 2014; DeYoung, 2015; Fleeson and Jayawickreme, 2015; Revelle and Condon, 2015).

The Structure of Individual Differences in Psychopathology

Psychiatric comorbidity is extensive in the general population (Kessler et al., 1994, 2005), and in clinical samples poly-diagnosis is the rule rather than the exception (Zimmerman and Mattia, 1999). This complicates clinical communication, treatment selection, and frustrates efforts to uncover the pathophysiology, etiology, and maintenance mechanisms of mental illness (Hyman, 2010). As a result, prominent clinical scientists, including the current and past heads of the U.S. National Institute of Mental Health, have called for a complete overhaul of the framework for classifying mental disorders (Hyman, 2010; Insel et al., 2010). Rather than enumerating increasingly detailed categories of disorder, it has been suggested that dimensions of functioning that cut across traditional diagnoses better approximate the structure of psychopathology (e.g., Brown et al., 1998; Widiger and Trull, 2007; Cuthbert and Insel, 2013; Harkness et al., 2014). One promising approach for addressing these issues involves statistically modeling patterns of covariation in diagnosed disorders and symptoms to clarify the natural between-person structure (BP-Structure) of mental disorders (Krueger and Markon, 2006; Wright and Zimmermann, 2015). This approach has been profitably applied to both child (Achenbach, 1966; Lahey et al., 2008) and adult (Krueger, 1999; Krueger and Markon, 2006; Kotov et al., 2011) disorders. In adult psychopathology, as noted above, a well-replicated BP-Structure has emerged based on individual differences in the clustering of disorders and their symptoms into Internalizing and Externalizing spectra (Wolf et al., 1988; Kotov et al., 2010a; Markon, 2010; Wright et al., 2013). This structure has demonstrated strong empirical and statistical evidence for its validity, including invariance across cultures (e.g., Slade and Watson, 2006), gender (Eaton et al., 2012), age groups (Eaton et al., 2011), and time-points within samples (Krueger et al., 1998; Vollebergh et al., 2001).

However, these domains are necessarily broad and decontextualized. In other words, they describe psychopathology in terms of individual differences, not in terms of the within-person or person-specific dynamic processes that often define mental disorders. Indeed, BP-Structural analyses of mental disorder covariation have largely relied on lifetime diagnoses (Krueger, 1999; Kotov et al., 2010a, 2011; Røysamb et al., 2011; Forbush and Watson, 2013; Wright et al., 2013) or some admixture of lifetime and current diagnoses (e.g., Markon, 2010; Blanco et al., 2013; Wright and Simms, 2015). What can be concluded from these studies is that the identified spectra of psychopathology (e.g., Internalizing, Externalizing) reflect latent dimensions of liability for the recognized mental disorders (Krueger and Markon, 2006; Caspi et al., 2014). That is to say, they reflect population-level risk for developing more specific instantiations of psychopathology during the lifespan. These spectra provide invaluable information about patterns of disorder covariation (i.e., co-morbidity), heritability (Kendler et al., 2011), and even the lack of specificity in responses to treatment (Barlow et al., 2010). Yet by themselves these dimensions lack the ability to provide information about proximal etiologies of clinically significant impairment, processes contributing to symptom exacerbation, or possible maintenance mechanisms.

Psychopathology as Maladaptive Dynamic Processes

Major theories of psychopathology posit processes of disorder development, exacerbation, and maintenance that play out over diverse time scales and frequently involve an interaction between individuals and the context in which they live their lives (e.g., Beck et al., 1979; Teasdale, 1988; Nolen-Hoeksema, 1991; Linehan, 1993; Benjamin, 2005). Indeed, many of the symptoms that define psychiatric disorders are cue- or context-dependent. For instance, social phobia is characterized by intense anxiety and behavioral avoidance when confronted with social or evaluative situations. The hallmark interpersonal impairments of borderline PD are responses to perceptions of significant others’ behavior. The binge-purge cycles of the patient diagnosed with bulimia nervosa reflect a maladaptive and extreme regulatory cycle (e.g., binges and purges both occur in response to heightened negative affect in a specific sequence). Even the blunted hedonic response in depression can be understood as a lack of the normative shift in affect in response to pleasurable events. This has led many researchers to begin studying the dynamic processes of psychopathology as they unfold in the naturalistic settings of daily life (e.g., Shiffman et al., 2002; Wegner et al., 2002; Silk et al., 2003; Ebner-Priemer et al., 2007; Trull et al., 2008; Sadikaj et al., 2013; Pe et al., 2015; see also Myin-Germeys et al., 2009 for a review).

This approach has provided much needed systematic empirical confirmation of the clinical description of psychiatric phenomena (e.g., affective instability in borderline PD; Russell et al., 2007; Trull et al., 2008) and has offered new insights into maladaptive behavioral sequences (e.g., individuals diagnosed with borderline PD are more likely to respond to perceived quarrelsomeness with negative affect, but no more likely to respond to negative affect with quarrelsomeness than controls; Sadikaj et al., 2013). Interestingly, as debates about psychiatric nosology have been pushing the field away from disorder-specific symptoms and toward dimensions that cut across traditional diagnoses, the study of dynamic processes in psychopathology has instead been emphasizing highly specific micro-processes (e.g., Wichers, 2014; Fried, 2015). It is notable that the majority, but not all, of the research studying dynamic processes in naturalistic settings have used a diagnostic group based design (e.g., comparing patients vs. community controls). Although there are plenty of good reasons for selecting circumscribed diagnostic groups for study (e.g., ensuring sufficient levels of pathology; maximizing statistical power in very expensive and difficult to collect data), this approach is at odds with efforts to collapse across categories to study dimensions of shared impairments (Krueger and Markon, 2006; Insel et al., 2010).

Thus, there is a tension between different areas of clinical science, which presumably share the same goal of clarifying the nature of psychopathology. The tension created is one between emphases on BP-Structure and within-person processes, which is certainly not a novel challenge (cf. Titchener, 1898). Taking as a given that both empirical thrusts have important information to contribute, the question becomes how best to integrate advances in the between-person structure of individual differences with the within-person study of dynamic processes (Wright, 2011; Hopwood et al., 2015).

Conceptually Integrating Between-Person Structure and Within-Person Dynamic Processes

A model for resolving the tension between investigations that focus on BP-Structure and within-person dynamic processes can be found in contemporary personality theory. Akin to the quantitative modeling of covariation in mental disorders, personality researchers invested heavily in the modeling of dispositional attributes that ultimately resulted in the Big-Five/Five-Factor Model of personality (for reviews see Digman, 1990, 1996; Goldberg, 1993; Wright, in press). Paralleling these investigations, researchers interested in personality processes have sought to study the within-person temporal dynamics of specific thoughts, feelings, and behavior, which are the behavioral building blocks of personality traits (e.g., Carver and Scheier, 1982; Larsen, 1987; Mischel and Shoda, 1995; Eid and Diener, 1999; Fleeson, 2001; Moskowitz and Zuroff, 2004; Cervone, 2005). Until recently and with few exceptions (e.g., Borkenau and Ostendorf, 1998), studies of the BP-Structure and dynamic processes of personality have largely proceeded separately (Read et al., 2010). There is now increasing interest in meaningful synthesis of models of individual differences in structure and the putative underlying dynamic processes that give rise to this structure (e.g., Fleeson, 2007; Fleeson and Gallagher, 2009; Fournier et al., 2009; DeYoung, 2015; Fleeson and Jayawickreme, 2015; Revelle and Condon, 2015). At the risk of oversimplifying, these integrative approaches take the domains outlined by BP-Structural models of individual differences (e.g., the Big-5), and use them as the orienting dimensions to organize hypotheses and investigations into the patterning of within-person dynamic processes (e.g., McCabe and Fleeson, 2012; Wright et al., 2015).

This integrative approach may be viable in psychopathology research given that the structures of personality and psychopathology are meaningfully overlapping (Wright and Simms, 2015). Long hypothesized, going back to antiquity and the writings of Hippocrates and Galen, evidence for the link between personality/temperament and mental disorders has is now quite robust. For one, several meta-analyses show that personality trait ratings and mental disorder diagnoses are strongly associated (e.g., Saulsman and Page, 2004; Ruiz et al., 2008; Samuel and Widiger, 2008; Kotov et al., 2010b). The meta-analytic results show that disorders falling within the Internalizing spectrum demonstrate strong associations with Neuroticism and Detachment (i.e., Introversion), whereas disorders falling within the Externalizing spectrum are most strongly associated with Disinhibition (i.e., low Conscientiousness and Impulsivity) and Antagonism (i.e., low Agreeableness).

Moreover, the hierarchical organization of personality traits and mental disorders bear unmistakable resemblance. A consistent finding is that at the level of two higher-order domains, dimensions of maladaptive personality bear close resemblance to the Internalizing and Externalizing spectra (Markon et al., 2005; Kushner et al., 2011; Wright et al., 2012; Wright and Simms, 2014). In these models, the Internalizing domain subsumes lower-order domains of Negative Affectivity and Detachment, and the Externalizing domain subsumes Disinhibition and Antagonism. Further, there is now accumulating evidence from models that incorporate broader sampling of psychopathology for additional spectra labeled Antagonism and Detachment/Anhedonic or Pathological Introversion (Markon, 2010; Kotov et al., 2011; Røysamb et al., 2011; Wright and Simms, 2015). Although direct evidence from hierarchical structural models of DSM diagnoses is lacking, the conceptual convergence with hierarchical models of personality suggests that a disorder based Antagonism domain can be joined with traditional indicators of disinhibitory pathology to form a broader Externalizing factor (e.g., Krueger et al., 2007), whereas Pathological Introversion would join affective disorders to define a higher order domain of Internalizing. Thus it is expected that with further targeted research the hierarchy of mental psychopathology and personality will largely converge.

Taken together, this suggests that much like contemporary personality science, investigations into within-person temporal processes of mental disorders could benefit from using the same empirically derived domains (e.g., Internalizing, Externalizing) that organize between-person differences in psychopathology in traditional cross-sectional research. As such, demonstrating similarities in structure at both levels would be the minimum requirement to ensure success of this approach. However, it is unknown whether the within-person structure (WP-Structure) that emerges from the temporal patterning of specific behaviors over time mirrors the BP-Structure of individual differences in the expression of those same maladaptive behaviors either at the higher-order level of Internalizing and Externalizing or possibly with lower-order differentiation of sub-factors within each domain.

Methodological Integration of Structure and Dynamic Processes

Joining models of between-person individual differences with the study of within-person dynamic processes immediately raises the issue of how to appropriately model and test whether such a marriage will succeed. Specifically, it involves modeling data that has a multilevel structure, with many time-points or occasions of measurement nested within individuals. There are two sources of variance in this type of data: variability associated with between-person differences in mean item endorsement, and variability associated with within-person, time-point specific deflections around those means. As Molenaar (2004) has shown, the structure of within-person covariation of behaviors is mathematically distinguishable from the covariation patterns of between-person differences in the mean levels of these behaviors (see also Nesselroade and Molenaar, 1999; Borsboom et al., 2003; von Eye and Bergman, 2003; Grice, 2004). That is to say, there is no guarantee that the same structure holds at both levels. Furthermore, the same WP-Structure may not apply to all individuals (Molenaar, 2004; Molenaar and Campbell, 2009). Indeed, for many applications, it is the person-specific (i.e., idiographic) structure (PS-Structure) that is of greatest interest. For instance, when it comes to tailoring and applying a behavioral intervention, substantial individual heterogeneity compels the development of a “model of the individual.”

A note on terminology is warranted. Here we draw a distinction between three tiers of structural analysis that are available when modeling intensive longitudinal data. BP-Structure refers to traditional conceptions of cross-sectional individual differences, and is derived from the covariation of behaviors averaged across time-points. Thus, it is time-invariant, or static in nature. We additionally consider two levels of dynamic structure. For the first dynamic approach, we use the term “WP-Structure” for the structure of temporal covariation in behaviors, pooled in whole or in part across individuals as is common in multilevel analysis. In other words, it is the within-person, dynamic patterning of behaviors, controlling for average levels, but shared, at least in part, across all individuals in the sample. For the second approach we use the term “PS-Structure” for person-specific models of temporal covariation that are based solely on a single subject’s multivariate time-series.

Several approaches have been developed for the appropriate structural analysis of intensive longitudinal data in groups of individuals. Multilevel structural equation modeling (MSEM; Muthén, 1991, 1994) generally offers a top–down approach, decomposing the total variance of the observed variables into the latent between- and within-person portions, and then fitting a model to each. It can be considered a top–down method because in MSEM a WP-Structure is specified that is then fitted to all individuals simultaneously (see also Shumway and Stoffer (2006) section 6.11 for a time-series perspective on this approach). Other methods adopt a bottom–up approach, starting with the structure of individuals and finding communalities in the individual data structures [e.g., the Integrated Trait-State Model (Hamaker et al., 2007) or Multilevel Simultaneous Component Analysis (Timmerman, 2006)], or iteratively fitting group- and individual-specific SEMs [e.g., Group Iterative Multiple Model Estimation (Gates and Molenaar, 2012)]. These approaches arrive at partially shared structure or parameters.

Methods for deriving PS-Structures involve the idiographic analysis of a single individual’s multivariate time-series. Certain methods are mathematically and conceptually parallel to the analysis of multivariate structure across individuals [e.g., P-technique Factor Analysis (Baldwin, 1946; Cattell, 1966)] or augment the analysis with temporal information by including a block-Toeplitz matrix [Dynamic Factor Analysis (Molenaar, 1985)] or using a multiple indicator vector autoregression moving average model (Hamaker et al., 2005). Recent developments include unified SEM (uSEM), which combines vector autoregression with SEM (Kim et al., 2007). Although similar, these methods differ in their emphasis on latent variables and inclusion of temporal lags [i.e., modeling associations from one time point (t-1) to the next (t)].

All of these methods share the ability to appropriately handle multilevel data structures, and each offers distinct advantages and disadvantages that need to be weighed with the specific modeling demands of the research question. To highlight a key distinction, the models of dynamic structure (i.e., WP-Structure or PS-Structure) differ in their level of complexity and flexibility in allowing for differences in structure across individuals. The least complex is the shared WP-Structure derived from MSEM with the most being the PS-Structure derived from idiographic analyses. Arguably, PS-Structure offers the most precise match to any given individual’s actual patterning of behavior. At the same time, when investigating large samples of individuals there may be value in using a more constrained approach like MSEM to appropriately reduce highly dimensional data into coherent but manageable factors. Practically speaking, investigators need to balance traditional assessment considerations (e.g., reliability of estimates, measurement error, bandwidth fidelity tradeoffs, etc.) with modeling complex nuanced dynamic processes.

The Current Study

The overarching goal of the current study was to provide a bridge between research paradigms that adopt divergent approaches to clarifying the fundamental nature of psychopathology. More precisely, we sought to provide a much needed conceptual link between work that has established transdiagnostic domains or crosscutting dimensions of psychopathology in cross-sectional data, and more recent efforts to understand the complex dynamic processes that characterize mental disorders as they play out in daily life. Toward this aim, we posed the following specific questions. First, does the BP-Structure of individual differences in daily endorsement of maladaptive behaviors conform to the latent structure of psychiatric diagnoses (i.e., Internalizing and Externalizing)? Second, can the same structure be applied to the WP-Structure of dynamic daily fluctuations in maladaptive behaviors? Third, is there PS-Structure heterogeneity in the daily and cross-day (i.e., lagged) links among the dimensions identified by WP-Structural analyses?

To answer the first two questions we estimated multilevel confirmatory factor analyses (i.e., MSEM) in a sample of individuals diagnosed with PDs who completed daily diaries of maladaptive behaviors over 100 consecutive days. We tested two a priori models for both the BP- and WP-Structures. We based our tested models on research on the overlap in BP-Structures of personality and psychopathology discussed above. Thus, we tested two-factor Internalizing and Externalizing models, as well as a nested four-factor model that partitioned Internalizing into its lower order domains of Negative Affectivity and Detachment, and partitioned Externalizing into lower order domains of Disinhibition and Hostility (representative of the broader Antagonism domain). Finally, we then addressed the third question by using uSEM to examine the person-specific interplay among components of this dynamic structure for a subset of participants, mapping networks in which Internalizing and Externalizing sub-factors influence each other at multiple temporal lags.

Materials and Methods

Participants

The sample used in this study was collected as part of a project designed to investigate general daily processes of behavior in individuals with PD. As such, recruitment targeted individuals diagnosed with any PD. Participants were recruited from a clinical sample (N = 628) enrolled in an ongoing study to improve efficient measurement of PD (Simms et al., 2011, under review). Participants were recruited into the broader clinical sample by distributing flyers at mental health clinics across Western New York, and were eligible for participation in the parent study if they reported psychiatric treatment within the past 2 years. Participants received structured clinical interviews by trained assessors for clinical syndromes and PDs using the sixth edition of the Mini International Neuropsychiatric Inventory (MINI; Sheehan and Lecrubier, 2010) and a version of the Structured Clinical Interview for DSM-IV-TR PDs (SCID-II; First et al., 1997), respectively. Only specific PD diagnoses were evaluated; PD-NOS was not evaluated or diagnosed. Disorder-level Kappas from independent ratings of a subset of participants (n = 120) were strong (Mdn K = 0.96; range = 0.66–1.00). Those who met the threshold for any PD diagnosis on the clinical interview were contacted for possible participation in the current daily diary study. The sole additional requirement for participation was daily Internet access via computer or mobile device.

One hundred and sixteen participants attended the baseline assessment for the daily diary study. Due to the focus on variability in behavior in this study, only participants providing at least 30 days worth of data were included to ensure reliable estimates of variability. Only 15 individuals were excluded for providing less than 30 diaries, resulting in an effective sample size of 101. Of these participants, 66 (65.3%) were female, and the majority reported being either White (82.2%) or African American (14.9%). On average, time between diagnostic interview and the initial assessment in this study was 1.4 years (Range = 1.2–1.7 years; SD = 0.16 years). The rates of PD diagnoses were as follows: 35.6% paranoid, 13.9% schizoid, 16.8% schizotypal, 7.9% antisocial, 36.6% borderline, 2.0% histrionic, 19.8% narcissistic, 53.5% avoidant, 5.9% dependent, 50.5% obsessive-compulsive. The average number of PD diagnoses per participant was 2.4. Additionally, 62.4% were diagnosed with mood disorders, 69.3% with anxiety disorders, 8.9% with psychotic disorders, and 23% with substance/alcohol use disorders. Demographics for the retained sample are presented in Table 1. Relative to the pool of 628 participants the current sample was drawn from, no differences were found on Age, Sex, or Employment Status. We found differences on Race (φ = 0.19), Marital Status (φ = 0.13), Educational Attainment (φ = 0.16), and Income (φ = 0.17), all of which were of small effect. Participants in the retained sample were less likely to be Black, and were more likely to be in higher Income or Educational Attainment categories. The retained sample was more likely to be married and less likely to be divorced or separated. Seventy-two percent of participants reported current mental health care treatment, 14% within the last year, and the remainder longer than 1 year prior to the daily diary protocol.

TABLE 1

TABLE 1. Sample demographics.

Procedure

A complete description of the study was provided, and written informed consent was obtained prior to participation in accordance with the Declaration of Helsinki. The University at Buffalo institutional review board approved all study procedures. Participants attended an initial in-person training and assessment session during which study procedures were explained, and a battery of self-report measures was completed via computer. Starting the evening of the in-person assessment, participants were asked to complete daily diaries assessing daily interpersonal behavior, affect, symptoms, stress, and functioning via secure website every evening for 100 consecutive days. Surveys were to be completed at roughly the same time each day, between 8 pm and 12 am. However, participants were allowed to deviate from this schedule if necessary (e.g., working nightshift) so long as (a) they completed diaries at the end of their day, and (b) the diaries were completed at roughly the same time each day. Participants received daily email reminders and also were provided several paper diaries they could use in the event of technological difficulties. Compliance rates were very high, with a total of 9,041 diaries completed by participants in this study after data cleaning (Mdn = 94 days, M = 89.5 days, range = 33–101 days, 90% > 60 days), a small fraction of which were done by paper (∼2% of completed diaries).¹ Compensation was provided for daily participation at the rate of $100 for ≥80% participation, and prorated at $1/day for <80%. Participation also was incentivized though recurring raffles ($10 drawing every 5 days for those providing at least four diaries) and drawings for additional money and tablet computers at the end of the study, with the odds of winning proportionally tied to participation.

Measures

Daily behaviors were measured using 16 items created for the purpose of this project. The specific questions used in this study are listed in the boxes denoting observed variables in Figure 1. These 16 items were selected for their relevance to the current study from a larger set of behaviors designed to provide broad coverage of the daily manifestations of personality pathology. Items were intended to reflect concrete behavioral manifestations of broad domains of personality pathology as they might occur in daily life. Items were written so that they were not so extreme as to have problematically low endorsement on a daily basis, and participants were given an 8-point response scale for each item anchored with Not at All (0) and Very Much So (7). Prior work in this sample has examined the basic descriptive features of these and the additional excluded items, including rates of endorsement, levels of (in)stability, and associations with the DSM-5 personality trait domains (Wright and Simms, under review). Of the 16 items used in this study, four were hypothesized primarily to reflect Negative Affectivity (multilevel coefficient alphas using Geldhof et al., 2014 approach were α_Between = 0.88, α_Within = 0.76), three primarily to reflect Detachment (α_Between = 0.84, α_Within = 0.64), four primarily to reflect Hostility (α_Between = 0.93, α_Within = 0.87), and the remaining five to primarily reflect Disinhibition (α_Between = 0.92, α_Within = 0.82).

FIGURE 1

FIGURE 1. Diagram of multilevel confirmatory factor analysis model with standardized factor loadings and latent covariances. Item specific residual variances and factor variances not depicted.

Additionally, symptom counts from diagnostic interviews for major depressive disorder, dysthymia, generalized anxiety disorder, social phobia, post-traumatic stress disorder, alcohol use disorder, substance use disorder, child conduct, antisocial, avoidant, dependent, borderline, narcissistic, histrionic, and paranoid PDs were used to develop a interview based structural model described below. Reliabilities (K’s) are reported above.

Data Analysis

As described in the introduction, our analyses incorporate both MSEM and uSEM, and we describe each in turn.

Testing BP- and WP-Structure of Maladaptive Daily Behaviors

The first two questions we sought to answer concerned whether the structure derived from individual differences in psychiatric diagnoses could be adequately fit to the daily diary data at the between- and within-person levels. We additionally sought to use the WP-Structure to reduce the dimensionality and complexity of the data for subsequent idiographic analysis. Therefore we selected MSEM as an analytic framework. MSEM extends traditional multilevel regression (i.e., hierarchical linear modeling; random coefficient regression) to multilevel covariance and mean structural modeling (Muthén, 1991, 1994). It does so by partitioning the total variance in the observed variables into the latent between-person variance (commonly referred between-cluster or between-group variance), and the observed within-person (also within-cluster or within-group) variance (Muthén, 1991). The partitioned variance can then be used to calculate both between- and within-person covariance matrices. Although the within-person covariance matrix is straightforwardly calculated and understood, calculation of the between-person covariance matrix is more complex (e.g., it is weighted for differences in cluster size) and is conceptually akin to the covariance among random intercepts (see Muthén, 1994 and Heck, 1999 for technical details, and Reise et al., 2005 and Preacher et al., 2010 for accessible summaries).

With the variance thus partitioned, MSEM offers the opportunity to separately estimate and compare between- and within-person structures by fitting standard latent variable models, like confirmatory factor analysis (CFA). A multilevel CFA was employed here and allows for potentially different factor structures to emerge at each level of the data. In the current context, the between-person structure reflects the pattern of covariation in average item endorsements over the course of the study, or, conceptually, the trait structure of these behaviors. In contrast, the within-person structure reflects the tendency for individual behaviors to covary at the daily level, or, conceptually, the dynamic structure of these behaviors. Here we estimate a series of between- and within-person factor models to determine the optimal structure of daily maladaptive behaviors sampled in this study.

For our first aim, we primarily were interested in testing whether a two-factor (Internalizing, Externalizing) model would acceptably fit the data, and if so, whether a four-factor model (Negative Affect, Detachment, Hostility, Disinhibition) improved upon this fit at the BP and WP levels. In addition to the 2- and 4-factor models of interest, we estimated one-factor models as a point of comparison. To test this, we estimated a series of MSEM models in Mplus version 7.31 (Muthén and Muthén, 1998–2012). Due to significant skew and kurtosis in the Externalizing behavior items, we treated all items as ordinal, and estimated multilevel CFAs using a robust (mean adjusted) weighted least squares approach (WLSM) on the polychoric correlation matrix. Model fit testing in MSEM can be challenging because the χ² test and alternative fit indices are derived from the comparison of the observed and implied covariance for both the between- and within-person matrices simultaneously. Therefore it is difficult to disentangle sources of ill model fit across levels. To address this complication, we adopted Ryu and West’s (2009) approach, which eliminates any source of ill fit from a given level by fitting a saturated model (i.e., zero df), while models of interest are tested in the other level. For example, first a saturated model was fit on the within-person level, and hypothesized models were fit to the between level, and then this process was reversed, fitting a fully saturated model at the between level, and estimating models of interest at the within-person level. Saturated models fit the data perfectly, and therefore they do not contribute to lack of fit, so any source of ill fit comes from the models at other levels. Ryu and West (2009) provide additional details about appropriate calculation of alternative fit indices for independence models at each level. Although evaluation of global model fit in MSEM remains an understudied topic, we considered the χ² test, as well as several alternative fit indices, using their single-level SEM recommended cutoffs (Hu and Bentler, 1999). These include the root mean square error of approximation (RMSEA) with values <0.05 for good model fit, comparative fit index (CFI) with values near or >0.95 indicative of good model fit, and the SRMR, with values <0.08 indicative of good model fit. Because we used the WLSM estimator, nested models were compared using the strictly positive Satorra–Bentler scaled χ² difference test (Satorra and Bentler, 2010).

In order to test the validity of our retained model, in a final MSEM we estimated interview-based Internalizing and Externalizing spectra using data from the structured clinical interview that were administered on initial assessment, and used these as predictors of the between-person factors from the daily behaviors. We used Kotov et al. (2011) structure, which includes PD diagnoses in the model, as a template to select relevant variables for our interview based model. We combined the diagnoses from Kotov et al.’s (2011) Externalizing and Antagonism domains in order to arrive at a broader Externalizing domain that would better match out daily behaviors. Thus, our interview-based Internalizing model was indicated by symptom counts for major depression, dysthymia, social phobia, post-traumatic stress, generalized anxiety, avoidant, dependent, and borderline PDs. The interview-based Externalizing factor was indicated by symptom counts for alcohol use, drug use, childhood conduct, adult antisocial, narcissistic, histrionic, paranoid, and borderline PDs (Please see Appendix A in Supplementary Material for example MSEM syntax).

Exploring Person-Specific Structures of Maladaptive Daily Behavior

Our second aim was to demonstrate how the optimal WP-Structure derived from the MSEM could be leveraged to inform person-oriented personality processes, providing a picture of the data at a third conceptual level of analysis. To accomplish this, we implemented individual-level unified structural equation modeling (uSEM; Kim et al., 2007; Gates et al., 2010). This approach combines SEM and vector autoregression of a single participant’s daily diary data in order to map the interplay among personality factors, that is, how variability in each factor is influenced by the contemporaneous (occurring on the same day, t) and lagged (occurring on previous days, t-1) variability in other factors. The model with a mean fixed zero is defined as:

$$\eta (\text{t}) = A\eta (\text{t})+{\Phi }_1\eta (\text{t} -1)+{\Phi }_2\eta (\text{t} -2)+\dots +{\Phi }_{\text{j}}\eta (\text{t} -\text{j})+\zeta (\text{t}),$$

where η(t) is the p-variate time series to be explained at day t = 1, 2,…, T, with p the number of MSEM-derived within-person factors and T the number of daily diary entries, A the (p,p)-dimensional matrix of contemporaneous regression coefficients explaining how each factor is influenced by other factors on the same day, Φ_q is the (p,p)-dimensional matrices of regression coefficients at lag q = 1, 2,…, j explaining how each factor is influenced by itself or other factors from previous days, and ζ is the p-variate error process, lacking sequential dependencies and having a zero mean and a diagonal contemporaneous covariance matrix. Simulation studies have found that incorporating contemporaneous and lagged effects simultaneously greatly improves reliability of results when compared to models that solely include one type of effect (Gates et al., 2010) and has been successfully applied to neuroimaging and observational data (Hillary et al., 2011; Beltz et al., 2013), with the present study being the first application to daily dairy data.

We fit uSEMs to the daily diary data of four exemplar participants in a data-driven fashion (cf. Gates et al., 2010) that accounted for the presence of multiple solutions (a characteristic of SEMs; MacCallum et al., 1993), and that satisfied the assumption of independent errors. We used LISREL for the analyses (Jöreskog and Sörbom, 1992). Model fitting was conducted in several steps. First, the items contributing to each MSEM within-person factor were averaged to create a factor composite score, as is commonly done in individual differences research. Second, a null uSEM model (i.e., no contemporaneous or lagged effects estimated) of the first order was fit to the data using the block Toeplitz method (cf. Molenaar, 1985). Third, Lagrange Multiplier tests (i.e., modification indices; Sörbom, 1989) were used to free and estimate the parameter in the A or Φ₁ matrix that would most improve model fit; this process iterated until no parameter would significantly (at p ≤ 0.05) improve model fit it if were freed. Multiple solutions could occur during this iterative process if modification indices showed that two parameters would equally improve model fit (i.e., their Lagrange Multiplier tests were equivalent). In these cases, each parameter was freed and estimated in a separate solution, and the iterative estimation process continued independently for each (with the possibility of further separations), generating a set of possible solutions (cf. Beltz and Molenaar, in revision). Fourth, non-significant parameters were trimmed from the models. Fifth, model fit was evaluated for the solutions using alternative fit indices, with two of the following four required to indicate excellent fit (Brown, 2006): RMSEA ≤0.05, SRMR ≤ 0.05, CFI ≥ 0.95, NNFI ≥ 0.95. Sixth, if multiple solutions occurred during the model fitting process, then the optimal solution was selected by choosing the model with the lowest AIC, a selection criterion employed in previous work (Akaike, 1974; MacCallum et al., 1993; Beltz and Molenaar, in revision). Seventh, the solution was examined for independent residuals using a posteriori model validation (Box and Jenkins, 1970). Specifically, one-step-ahead prediction errors were generated from the model and tested for white noise (cf. Beltz and Molenaar, 2015). If white noise was found, then a first order uSEM was appropriate for the data, and the solution was accepted. If white noise was not found, then a first order solution was insufficient for capturing all sequential dependencies in the data, and steps two through seven were repeated for a second order uSEM (i.e., a model with A, Φ₁, and Φ₂ matrices; Please see Appendix B in Supplementary Material for example uSEM synt).

Results

Global model fit and model fit comparisons for the MSEM analyses can be found in Table 2. Starting with the between-person level, all estimated models were considered a good fit to the data using the chi-square tests, which were uniformly non-significant. This was expected due to the low-powered test with a between-person sample size of 101. The RMSEAs and CFIs also were excellent, although the SRMR was only acceptable in models with 2 and 4 factors. The likelihood ratio test indicated that model fit improved going from 1 to 2 factors, but a four-factor model did not significantly improve the fit. As such, we selected a two-factor structure as the optimal BP model in these data. For the WP level, the chi-square tests were uniformly significant. This was expected due to the high-powered test with 9,041 within-person observations. The RMSEA and CFI suggested all models were good fitting, although each improved appreciably going from 1 to 4 factors. The SRMR was only acceptable in models with 2 and 4 factors. Finally, the chi-square difference test strongly favored a four-factor solution. Thus, our final retained model differed in structure across levels of analysis, with two factors at the between-person level, and 4 factors at the within-person level of analysis.² The model with standardized parameter estimates can be found in Figure 1.

TABLE 2

TABLE 2. Multilevel confirmatory factor analysis model fit and model fit comparisons.

To test the validity of this model, we estimated Internalizing and Externalizing factors from the original clinical interviews and regressed the daily diary based Internalizing and Externalizing factors on each of these. This resulted in an excellently fitting model [χ²₍₅₂₅₎ = 568.54, p = 0.09; RMSEA = 0.003; CFI = 1.00; NNFI = 1.00; SRMR_Within = 0.05, SRMR_Between = 0.09]. Relevant model parameter estimates can be found in Figure 2. We found that the interview-based Internalizing factor was a significant predictor of daily Internalizing (β = 0.60; 95% confidence interval = 0.38 to 0.81; p < 0.001), but not of daily Externalizing (β = 0.15; 95% confidence interval = -0.09 to 0.38; p < 0.22). In contrast, we found that the interview-based Externalizing factor was a significant predictor of daily Externalizing (β = .41; 95% confidence interval = 0.18 to 0.63; p < 0.001), but not of daily Internalizing (β = -0.08; 95% confidence interval = -0.32 to 0.16; p < 0.55). Thus, our between-person factors estimated from daily diaries evidence significant associations with corresponding traditional interview based factors, and these associations were specific.

FIGURE 2

FIGURE 2. Diagram of multilevel structural equation model (standardized estimates) with interview based Internalizing and Externalizing factors predicting daily diary based Internalizing and Externalizing Spectra. Item specific residual variances and factor variances not depicted. Asterisks depict significant regression paths between latent factors (^∗∗∗p < 0.001), but significance is not provided for other parameters.

Next, we examined person-oriented personality processes by mapping with uSEM the interplay among the four within-person factors for four exemplar participants; four, three, five, and four items as indicated from the MSEM analysis were averaged to create the Negative Affect, Detachment, Disinhibition, and Hostility composite scores, respectively; see Figure 1. Time series plots and descriptive statistics for each participant’s scores are shown in Figure 3. Notice how some characteristics of participants’ daily responses shown in the time series plots were independent of the descriptive statistics (i.e., means and standard deviations shown in the bar graphs). For example, participants A, B, and C had similar Disinhibition means and standard deviations despite markedly different patterns of responses across days, such as participant B having large peaks and valleys that appear to co-occur with Negative Affect, a pattern not seen in the others. Also, participant D had means close to zero for all composite scores even though Detachment scores were close to 5 on a couple of days. Finally, participant C had long periods of constant Detachment scores, but this information is lost in the descriptives. These are precisely the characteristics – those typically lost in cross-sectional research or when time series data are analyzed in aggregate – that uSEM captures and reflects in individual-level dynamic personality network maps, revealing the person-oriented processes underlying personality.

FIGURE 3

FIGURE 3. Time series plots and descriptive statistics for composite scores of the four within-person factors used in person-oriented uSEM analyses for each of four exemplar participants (A–D). Time series plots show the composite scores for each daily diary. Bar graphs show the means and standard deviations of the scores across all diaries. Within-person factors in shades of blue define the between-person factor of internalizing, and within-person factors in shades of yellow define the between-person factor of externalizing.

Model fit can be found in Table 3, and the final uSEM maps for each participant are shown in Figure 4. The final map for exemplar participant A (Figure 4A) fit the data well; multiple solutions were not present, and first order relations were sufficient for capturing all sequential dependencies in the data. The final map for exemplar participant B (Figure 4B) fit the data well; four solutions were generated, with the retained solution selected based on lowest AIC and first order relations were sufficient. The final map for exemplar participant C (Figure 4C) fit the data well; seven solutions were generated, AIC was again used to select the final solution, and first order relations were sufficient. The final map for exemplar participant D (Figure 4D) fit the data well; three solutions were generated, and second order relations were required to capture all sequential dependencies in the data.

TABLE 3

TABLE 3. Unified SEM (uSEM) model fit results for four exemplar participants.

FIGURE 4

FIGURE 4. Network maps from uSEM analyses, showing the interplay among the four within-person personality factors for each of four exemplar participants (A–D). Solid lines are contemporaneous relations, dashed lines are relations of the first order, dotted lines are relations of the second order, black lines are positive relations, red lines are negative relations, and beta-weights show the magnitude of the relations (all significant at p ≤ 0.05). All maps fit the data well; see fit statistics in text.

The maps can be understood as visual depictions of a series of regression equations (consistent with the beta-weights that accompany the relations), with one equation for each personality factor. A simple example concerns the Detachment of participant B: on any given day, it was positively predicted by Negative Affect occurring on the same day, meaning that increases (decreases) in Negative Affect statistically predicted increases (decreases) in Detachment. This is consistent with the synchronous rise and fall of Negative Affect and Detachment scores visible in the time series plot. A more complex example concerns the Negative Affect of participant A: on any given day, it was explained by Negative Affect (i.e., itself) levels from the previous day, and inversely by Detachment levels from the previous day. Participant A’s Negative Affect was also explained by Hostility and Detachment levels on the same day, and it predicted Detachment levels on the same day as well as Hostility levels on the next day. The set of relations between Negative Affect and Detachment (with reciprocal same day relations, and an inverse prediction of Negative Affect by Detachment) and the pair of relations between Negative Affect and Hostility (with Negative Affect predicting Hostility on the next day, but Hostility predicting Negative Affect on the same day) suggest the presence of feed-forward and feedback mechanisms.

The maps reveal several interesting findings. Visual inspection shows different personality dynamics for each of the participants. For example, the map for participant B was the sparsest (i.e., had the fewest relations with 6), and the maps for the other three participants were equally dense (i.e., had 10 relations). This is an especially interesting finding for participant D, who had composite scores close to zero and second order map relations, suggesting the presence of complex personality processes despite low mean levels of endorsed symptomatology. Graph theoretical metrics, such as total degree (i.e., the number of incoming and outgoing relations for a factor), reveal that Negative Affect was the most important factor for participants A and B, Detachment and Hostility were most important for participant C, and Disinhibition was most important for participant D. This is intuitive in some cases (e.g., Negative Affect also had the highest mean for participant A), but not in others (e.g., Detachment had long periods of constant scores and Hostility had the lowest mean for participant C).

Discussion

The overarching aim of our study was to provide a conceptual and analytic integration of individual differences research on the structure of psychopathology liability and the complex dynamic processes that comprise mental disorders. First, using data from a sample of individuals diagnosed with PDs who completed several months of (Mdn observation N = 94) daily diary studies, we tested the BP- and WP-Structures of 16 behaviors chosen to index the broader Internalizing and Externalizing spectra with MSEM (see Figure 1). Results indicated that for the BP-Structure, a two-factor model (Internalizing and Externalizing) was sufficient for explaining individual differences in the endorsement of the 16 behaviors, whereas for the WP-Structure a more differentiated four-factor model was supported (Negative Affect, Detachment, Disinhibition, and Hostility). We then demonstrated that the BP-Structure of this model showed specific associations with similar factors derived from traditional psychiatric interviews. Second, using the results of the MSEM, which served to greatly reduce the multivariate modeling space from the individual items to these four factors, we fit a set of idiographic uSEM models to a subset of the participants (n = 4) in order to showcase the heterogeneity in dynamic associations among the four daily constructs (i.e., PS-Structures). Thus, our approach represents a hybrid of using MSEM for data reduction and theoretical model testing, followed by the data driven exploration of fine-grained person-specific dynamic processes. We consider each analytic approach and set of results in turn.

Integrating Structure and Process: Daily Internalizing and Externalizing Behaviors

This study was motivated, in large part, by a tension that has developed in the science of psychopathology; namely, how can models that seek to establish crosscutting dimensions of functioning be reconciled with data collection and analytic approaches that seek to study nuanced contextualized processes? As noted in the introduction, this is a basic tension that has long existed in the personality literature (e.g., Read et al., 2010), which only lately has been given serious theoretical attention (e.g., Fleeson and Jayawickreme, 2015). Adopting some of the conceptual strategies from basic personality science, we tested whether the structure of daily fluctuations in maladaptive behavior conformed to a similar structure derived from individual differences in lifetime psychiatric diagnosis; specifically, the Internalizing and Externalizing spectra. We found that a variant of the hypothesized structure provided a good fit to the data. At the between-person level the items mapped onto a clear two-Factor Internalizing-Externalizing structure. However, the correlation among these two factors was high (r = 0.77), indicating that those individuals who report higher average levels of daily Internalizing behavior also report higher average levels of daily Externalizing behavior. Prior research has, in fact, found correlations among lifetime variants of these two factors ranging from modest to strong (range of rs = 0.17–0.56) (e.g., Krueger, 1999; Markon, 2010; Røysamb et al., 2011; Forbush and Watson, 2013; Kotov et al., 2011; Wright et al., 2013). Although our final model would suggest these two domains are significantly discriminable, their covariation argues for the importance of considering general severity in daily psychopathology (cf. Kessler et al., 2005; Caspi et al., 2014). Due to the novelty of these analyses, direct comparisons with additional samples are not possible. As such it remains unclear why the covariation between these two factors here is higher than in traditional individual differences work. It may be due to truly higher overlap among daily behaviors, the manner in which MSEM partials between-person variance, the estimator (i.e., robust WLS), the severe nature of the sample, or other factors. Future research will be needed to clarify the degree of overlap among individual differences in these domains derived at the daily level.

More central to our aim, we found that the WP-Structure of daily maladaptive behaviors was more differentiated than the BP-Structure. The identical two-factor Internalizing-Externalizing structure as the between-person model provided good fit to the data by most indices, even as the four-factor structure provided significantly improved fit. The fanning out of content at the four-factor model is consistent with structural models of psychopathology (Markon, 2010; Røysamb et al., 2011; Wright and Simms, 2015) and maladaptive personality traits (Markon et al., 2005; Kushner et al., 2011; Wright et al., 2012; Wright and Simms, 2014) and can be understood as more circumscribed variants of the “pathological Big-4” (Livesley et al., 1998; Widiger and Simonsen, 2005; Calabrese et al., 2012). As such, the resultant 2- and 4-factor structures reflect the hierarchical organization of personality and psychopathology.

It is difficult to overstate the importance of understanding these dimensions as hierarchically organized. In the contem porary era of studying contextualized processes, as researchers seek to study putatively highly specific dynamic phenomena, there will be a pressing need to organize the results of individual studies, highlighting near-neighbor processes for investigation and mapping out “dynamic nomological nets” (cf. Cronbach and Meehl, 1955). As specific dynamic processes are proposed and tested, it will behoove researchers to test for convergent and discriminant validity in near neighbor constructs. For instance, hypotheses that specify processes associated with negative affect should demonstrate discriminability between specific affects and/or detachment related processes. This is a basic approach adopted in individual differences research, and will serve to further clarify specificity and generality in dynamic processes in psychopathological research.

Estimating and Interpreting Individual uSEM Models and Treatment Implications

Armed with the reduced dimensionality of the four-factor WP-Structure, we then sought to demonstrate that it provides a strong platform for studying dynamic PS-Structures as they play out across days. We approached this by estimating uSEM models at the individual level for four exemplar participants; we mapped the lagged and contemporaneous interplay among Negative Affect, Detachment, Disinhibition, and Hostility for each person. The heterogeneity among these participants is evident in their time series plots (Figure 3) and in their network maps (Figure 4). The time series showed that each individual tracked different Negative Affect, Detachment, Disinhibition, and Hostility trajectories. For example, participants A and D had Detachment ratings that were mostly low and punctuated by relatively few extreme spikes, whereas participants B and C had Detachment ratings that were constantly changing. The maps showed a highly distinct network of associations among the domains of pathology for each individual. For example, the relations between Negative Affect and Detachment differed among the participants, with Negative Affect positively predicting Detachment for participants B and C, reciprocal positive contemporaneous relations between Detachment and Negative Affect on the same day and Detachment inversely predicting Negative Affect on the next day (perhaps evidencing feed forward and feedback loops) for participant A, and no association between the behaviors for participant D.

In the context of a research study, each of these network maps catalyzes the imagination, leading to questions about how these processes play out in participants’ daily lives. However, in different contexts, specifically clinical settings, one could envision collecting similar data, and using these models to develop hypotheses about a patient’s particular sequence of maladaptive behavior and points of intervention. Take, for example, the uSEM model for participant A. The model suggests that Negative Affect is the lynchpin in this individual’s pathology. Negative Affect, although central to the person’s structure, is more often than not an outcome. Thus, an initial point of intervention may be to address predictors of this individual’s Negative Affect, such as Detachment and Hostility. Hostility, for example, appears to drive same day Negative Affect and Disinhibition, which leads to decreases in next day Detachment, perhaps suggesting pursuing rapprochement with embattled others. This may signal a relative interpersonal strength or healthy functioning that might be leveraged in a treatment. Many additional distinct hypotheses flow from examining the remaining paths across the four maps.

Clinically, the goal would be to disrupt these processes in order to effect change. Yet it would not require distal armchair speculation, as the clinician and patient would have proximal experience with which to augment these quantitative findings. A practitioner could use similar diagrams to those presented here as a tool to engage the patient in a collaborative discussion of how he or she understood his or her own processes, and together develop a target and plan for intervention. This approach of developing hypotheses based on coefficients derived from intensively sampled data and integrating the patient’s own phenomenology is likely viable, as similar methods have been furthered and tested based on traditional dispositional measures (e.g., Finn, 2007). Thus, these results have the potential for direct clinical applicability, at least as a novel tool that can be taken from bench to bedside. In fact, several of the modeling challenges (e.g., multiple well fitting solutions; see Beltz and Molenaar, in revision) may be seen as a boon because they can be presented as alternative hypotheses for the patient to choose from, thereby engaging him or her in his or her treatment. The major rate-limiting step is the development and dissemination of powerful but user-friendly data collection tools, analysis software, and research on the use in clinical practice.

Selecting an Appropriate Modeling Framework and Alternative Approaches

Refocusing our lens on the methods, we note that there are always a number of decision points to navigate when doing any statistical modeling. With highly multivariate, intensive, longitudinal data across many individuals, the number of possibilities for different analytic approaches is, to say the least, quite large. There were several major considerations that we grappled with, of which we mention two here: (1) adopting a confirmatory vs. exploratory framework, and (2) deciding whether to estimate structures with parameters that varied or were shared across individuals.

First, both confirmatory and exploratory models can be estimated in a MSEM framework. Our primary goal here was to test the degree to which an established model could be fit to a distinct data type. However, different modeling scenarios may compel an exploratory framework. Much needed is basic psychometric and scale development work for item banks to be used in intensive longitudinal data. The same degree of care that has been put into cross-sectional measures has generally not been incorporated in the measures used in dynamic processes (for an exception see Tomko et al., 2014).

Second, one of the exciting possibilities afforded by intensive longitudinal data of the type we modeled here, is that it allows for the estimation of not only the structure of individual differences, but individual differences in structure (i.e., idiographic structures). A challenge for covariance-based idiographic modeling approaches such as uSEM is that they require a minimum of variance in each observed variable in order to be included in the estimation (cf. Nesselroade et al., 2007). In prior studies, which have used many fewer participants, large portions of items have had to be discarded due to lack of endorsement (Nesselroade et al., 2007). This becomes particularly problematic given the use of maladaptive items, which tend to have lower endorsement, even among clinical samples. Even considering the 16 items used here, for many of the individuals specific items would have to be discarded. Therefore, by using MSEM (which estimates the WP-Structure pooled across individuals) or by creating behavioral composites for idiographic analyses, all participants and items can be included in the model.

A related consideration is the optimal degree of complexity for the estimated networks of dynamic processes. For some applications, very specific behaviors may be desired (e.g., in the study of suicidal attempts), but as granularity increases, so too does the potential network complexity. To make this concrete, consider assessing 20 specific negative affect items at each assessment, and analyzing a network of associations among the individual items. This would result in up to 380 possible contemporaneous associations, not considering lagged associations. This would strain direct interpretability, and place limits on the amount of other domains (e.g., social behavior, cognition, motivation) that could be modeled congruently. Naturally, graph theory indices (e.g., node centrality) can be used to winnow down such a highly parameterized model. Alternatively, selecting fewer but broader domains offers desirable qualities like enhanced reliability of assessment, greater bandwidth of measurement, and easier interpretability. The point is that researchers need to be mindful of the optimal level of granularity for their questions of interest.

Limitations and Future Directions

Several limitations with the current study bear mention. For one, our model derives from the specific set of daily behaviors we chose to measure, and it does not include items related to daily substance use. This is a potential limitation seeing as substance abuse forms a major component of the traditional dispositionally estimated Externalizing domain. Nevertheless, substance abuse is thought to reflect specific instantiations of broader constructs such as Disinhibition or impulsivity, which were well covered in our daily diary data. This is evident in the significant regression path between our interview and daily diary based Externalizing factors. Additionally, the results must be interpreted in the context of this specific sample, which was not a random section of the population, but rather selected to possess elevated psychopathology. Specifically, the current sample was selected for a diagnosis of any PD. Although this ensured breadth of psychiatric diagnoses due to well-established comorbidity patterns, and participants additionally met the criteria for several other clinical syndromes (e.g., anxiety disorders, mood disorders, substance use disorders), future work would benefit from a broader range of severity.

Furthermore, we only estimated and presented uSEM models for four participants. As noted above, this was primarily to demonstrate that, despite strong covariation among factors in the WP-Structural model, individuals exhibit rich and interesting heterogeneity in the dynamic processes constituting that model; the factors within the WP-Structure have contemporaneous and lagged associations with each other that are directional and unique to each participant. Future work should examine the full sample, ideally with a method that can establish shared and unique pathways across individuals (e.g., GIMME; Gates and Molenaar, 2012).

A related issue is that these models were estimated as context independent, and future work is needed that incorporates external variables (e.g., daily stress; cf. Gates et al., 2011). In the current sample we additionally measured a variety of daily stressors, perceived stress in response, several indicators of daily functioning (e.g., sleep, job attendance), and a number of basic behaviors (e.g., affect, social behavior) in the daily diary. In subsequent investigations we plan to examine daily stressors as a contextual input to the system predicting daily fluctuation in psychopathology domains. Similarly, we hope to examine the effect of fluctuations in daily psychopathology on daily functioning. Finally, we hope to test whether a variety of baseline dispositional assessments serve amplify or dampen these within-person linkages in context (stress), psychopathology, and functioning.

An Agenda for Integrating Empirical Structure and Dynamic Processes in Psychopathology

As this study is the first to attempt to bridge the contemporary empirical thrusts of structural and dynamic investigations into psychopathology, we have merely scratched the surface of what is possible. Here we outline several necessary steps toward more fully realizing the potential of an integrative science of psychopathology.

First, as mentioned briefly above, we must stress the need for measurement development and normative data collection. The items used here were developed ad hoc for this current project as no inventory for intensive repeated measurement of psychopathology (e.g., momentary, daily, etc.) was available. That these items performed extremely well as intended is very encouraging, but that should not preclude more extensive measurement development and refinement. Structural models of individual differences, which have arguably established broad domains of relevant phenotypic functioning (Harkness et al., 2014), provide a firm base from which to launch these measure development excursions. Moreover, there are many other variables that could imbue this work with more psychological texture and nuance. For instance, incorporating motivations and goals in addition to behavior and affect would likely prove fruitful. Going hand in hand with this effort should be the collection of normative data. If intensive repeated measurement is to be used clinically, then established norms, in both the population and treatment samples will be necessary. This will involve more than just importing and applying traditional psychometrics (e.g., means), but the thoughtful application of existing and development of novel “dynamic psychometrics.” At the most basic level this might include relevant measures of variability and instability of behavior over time (e.g., Jahng et al., 2008; Houben et al., 2015), but should conceivably be expanded to include normative associations between daily behaviors, behaviors and environmental antecedents (e.g., What is the average strength of association between daily stress and hostility? Or, between social anhedonia and withdrawal?). Integration of graph theory metrics into the normative data description may also prove fruitful, especially as variable sets increase in number and complexity.

Second, we have focused here on the individual as a closed system, considering neither environmental inputs nor impact on external variables (e.g., other people). However, it is well understood that humans are not closed systems, and indeed as reviewed briefly above contemporary theories of psychopathology are largely based on models of the individual acting in context. Thus, further research in this vein should incorporate traditional inputs and outputs to the system in the form of putative environmental antecedents, stressors, protective factors, and functional concomitants of maladaptive functioning. At the same time we underscore that although traditional perspectives might draw distinctions between environmental and individual located variables (e.g., Cohen et al., 1995) in practice these distinctions are difficult to make, and the current diagnostic nomenclature blends contextual, behavioral, and functional variables. In this regard, the focus on and use of more fine-grained data sampling and analytic approaches may help disentangle the problematically heterogeneous disorder based models of psychopathology.

From this follows our third suggestion, that further work in this area should move forward unencumbered by traditional diagnostic categories. Current models of psychopathology reflect top–down organizational schemes, and are largely studied as such. In the current study we have similarly adopted a top-down perspective in part, in that we used results from quantitative structural models of psychopathology as the starting point for establishing structural models of daily behavior. We believe this provides useful if not necessary scaffolding for the subsequent person-specific analyses. Yet whether one starts with a refined (as we suggest) or an unconstrained set of variables, starting from the bottom–up and seeking out individual differences in dynamic patterns of behavior is an avenue ripe for exploration; especially if combined with techniques that can establish relatively homogenous groupings of individuals based on shared parameters. Building on this, there is a need for more research that takes intensive repeated measurement as a starting point, and seeks to establish functional outcomes that are strongly if not uniquely predicted by resulting parameters (e.g., Stepp et al., 2011; Forbes et al., 2012).

Finally, we believe that this line of research is ideally suited to refining the way in which clinicians assess, diagnose, and monitor treatment effects. For one, the approach is rooted in a dimensional architecture that recognizes that psychopathology varies along gradients of severity, and does not adhere to convenient but arbitrary boundaries. But key is that it incorporates that psychopathology is a process, and therefore should be assessed as such. Thus beyond goals of reducing overall symptom levels, many of the processes hypothesized to drive change in psychotherapy involve not only the decrease in the level of one variable, but also the dynamic change in the association among multiple variables. Indeed, as we have been arguing, psychological symptoms rarely occur in isolation, and instead are coupled with specific contingencies, linked with problematic behaviors, and connected with maladaptive processes. As a result, clinicians often seek not just to decrease a problematic behavior, but also to change the connection between two or more behaviors in order to disrupt the maladaptive processes that maintain psychopathology. Examples include increasing emotional differentiation (i.e., unlink distinct negative emotions), diminishing the link between negative emotions and maladaptive self-regulation (e.g., cutting, substance use, withdrawal), increase positive coping behaviors when distressed, increase tolerance of anxiety in feared situations, and attenuating the link between triggering stimuli and phobic responses. Quantitatively, each of these would be represented by a dynamic relationship among variables—or, stated otherwise, an association that changes over time. Accordingly, the targeted variables should be assessed in a manner that allows for establishing the strength of these links, and then repeatedly assessed in a way that allows for the continued probing of the strength of that link via appropriate quantitative methods (Wright et al., 2014).

The avenues for future work in this area are wide open, and we have outlined but a few potential directions for a program of research that seeks to integrate structural and dynamic processes. Importantly, much of the fundamental work has yet to be done, starting with measurement, establishing structural similarities, refining the psychometrics, and then moving from a well-established beachhead into more complex and nuanced investigations.

Conclusion

In sum, we believe the findings presented here are an initial step down one possible path toward merging two contemporary paradigms in psychopathology research, the psychometric approach to establishing crosscutting domains, and the investigation of contextualized dynamic processes. It is our hope that the clearly interpretable factor solutions estimated at the between- and within-person levels demonstrate that domains derived from the study of individual differences in psychopathology can be fruitfully applied and used to organize investigation into person-specific dynamic processes. This approach is already being implemented in basic and applied personality science to establish contingencies and mechanisms driving behavior (e.g., Beckmann et al., 2010; McCabe and Fleeson, 2012), and similar approaches should be viable in psychopathology research.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This research was supported by the National Institute of Mental Health (F32 MH097325, L30 MH101760, Wright; R01 MH080086, Simms), National Institute of Biomedical Imaging and Bioengineering (R21 EB015573, Gates) and the National Science Foundation (NSF 1157220, Molenaar). The views contained are solely those of the authors and do not necessarily reflect those of the funding source.

Supplementary Material

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01914

Footnotes

1. ^ We examined basic demographics of gender and age, severity of personality disorder based on clinical interview, and average reported daily diary domains that we examined in the study. We found modest correlations of rate of participation with age (r = 0.20, p = 0.04) but not gender (r = 0.17, p = 0.10), no association with any of the personality disorders (all p’s > 0.10), and modest associations with average daily hostility (r = -0.24, p = 0.02) and disinhibition (r = -0.23, p = 0.02) but not the remaining two daily domains (p’s > 0.07). Thus, individuals who were younger and reported higher levels of daily Externalizing behavior participated less. However, we believe this had little influence on the results given that (a) these associations are in the context of very high rates of participation (Mdn = 94 days; M = 89.5 days) and (b) multilevel SEM weights participants contribution to the covariance matrices based on the number of observations.
2. ^ To be comprehensive, we estimated three-factor models at each level, which had a negligible impact on fit at the between-person level, but resulted in significantly poorer fit relative to the four-factor model at the within-person level. We additionally note that changing the estimator to robust maximum likelihood (MLR) resulted in identical conclusions, as did mean and variance adjusted weighted least squares (WLSMV) estimation. In fact, all parameter estimates were identical using the WLSMV estimator, and model fit was substantially improved, but direct comparison of fit cannot be accommodated in a MSEM framework in MPlus. Thus, our results and conclusions are robust to estimation approach.

References

Achenbach, T. M. (1966). The classification of children’s psychiatric symptoms: a factor-analytic study. Psychol. Monogr. Gen. Appl. 80, 1–37. doi: 10.1037/h0093906

CrossRef Full Text | Google Scholar

Akaike, H. (1974). A new look at the statistical model identification. IEEE Trans. Autom. Control 19, 716–723. doi: 10.1109/TAC.1974.1100705

CrossRef Full Text | Google Scholar

Baldwin, A. L. (1946). The study of individual personality by means of the intraindividual correlation. J. Pers. 14, 151–168. doi: 10.1111/j.1467-6494.1946.tb01044.x

CrossRef Full Text | Google Scholar

Barlow, D. H., Farchione, T. J., Fairholme, C. P., Ellard, K. K., Boisseau, C. L., Allen, L. B., et al. (2010). Unified Protocol for Transdiagnostic Treatment of Emotional Disorders: Therapist Guide. Kettering, OH: Oxford University Press.

Google Scholar

Beck, A. T., Rush, A. J., Shaw, B. F., and Emery, G. (1979). Cognitive Therapy of Depression. New York, NY: Guilford Press.

Google Scholar

Beckmann, N., Wood, R. E., and Minbashian, A. (2010). It depends how you look at it: on the relationship between neuroticism and conscientiousness at the within-and the between-person levels of analysis. J. Res. Pers. 44, 593–601. doi: 10.1016/j.jrp.2010.07.004

CrossRef Full Text | Google Scholar

Beltz, A. M., Beekman, C., Molenaar, P. C. M., and Buss, K. A. (2013). Mapping temporal dynamics in social interactions with unified structural equation modeling: a description and demonstration revealing time-dependent sex differences in play behavior. Appl. Dev. Sci. 17, 152–168. doi: 10.1080/10888691.2013.805953

PubMed Abstract | CrossRef Full Text | Google Scholar

Beltz, A. M., and Molenaar, P. C. M. (2015). A posteriori model validation for the temporal order of directed functional connectivity maps. Front. Neurosci. 9:304. doi: 10.3389/fnins.2015.00304

PubMed Abstract | CrossRef Full Text | Google Scholar

Benjamin, L. S. (2005). “Interpersonal theory of personality disorders: the structural analysis of social behavior and interpersonal reconstructive therapy,” in Major Theories of Personality Disorder, 2nd Edn, eds M. F. Lenzenweger and J. Clarkin (New York, NY: Guilford Press), 157–230.

PubMed Abstract | Google Scholar

Blanco, C., Krueger, R. F., Hasin, D. S., Liu, S. M., Wang, S., Kerridge, B. T., et al. (2013). Mapping common psychiatric disorders: structure and predictive validity in the national epidemiologic survey on alcohol and related conditions. JAMA Psychiatry 70, 199–207. doi: 10.1001/jamapsychiatry.2013.281

PubMed Abstract | CrossRef Full Text | Google Scholar

Borkenau, P., and Ostendorf, F. (1998). The big five as states: how useful is the five-factor model to describe intraindividual variations over time? J. Res. Pers. 32, 202–221. doi: 10.1006/jrpe.1997.2206

CrossRef Full Text | Google Scholar

Borsboom, D., Cramer, A. O., Schmittmann, V. D., Epskamp, S., and Waldorp, L. J. (2011). The small world of psychopathology. PLoS ONE 6:e27407. doi: 10.1371/journal.pone.0027407

PubMed Abstract | CrossRef Full Text | Google Scholar

Borsboom, D., Mellenbergh, G. J., and van Heerden, J. (2003). The theoretical status of latent variables. Psychol. Rev. 110, 203–219. doi: 10.1037/0033-295X.110.2.203

CrossRef Full Text | Google Scholar

Box, G. E. P., and Jenkins, G. M. (1970). Time Series Analysis: Forecasting and Control. San Francisco, CA: Holden-Day.

Google Scholar

Brown, T. A. (2006). Confirmatory Factor Analysis for Applied Research. New York, NY: Guilford Press.

Google Scholar

Brown, T. A., Chorpita, B. F., and Barlow, D. H. (1998). Structural relationships among dimensions of the DSM-IV anxiety and mood disorders and dimensions of negative affect, positive affect, and autonomic arousal. J. Abnorm. Psychol. 107, 179–192. doi: 10.1037/0021-843X.107.2.179

PubMed Abstract | CrossRef Full Text | Google Scholar

Calabrese, W. R., Rudick, M. M., Simms, L. J., and Anna Clark, L. (2012). Development and validation of big four personality scales for the schedule for nonadaptive and adaptive personality—second edition (SNAP-2). Psychol. Assess. 24, 751–763. doi: 10.1037/a0026915

PubMed Abstract | CrossRef Full Text | Google Scholar

Carver, C. S., and Scheier, M. F. (1982). Control theory: a useful conceptual framework for personality-social, clinical, and health psychology. Psychol. Bull. 92, 111–135. doi: 10.1037/0033-2909.92.1.111

PubMed Abstract | CrossRef Full Text | Google Scholar

Caspi, A., Houts, R. M., Belsky, D. W., Goldman-Mellor, S. J., Harrington, H., Israel, S., et al. (2014). The p factor one general psychopathology factor in the structure of psychiatric disorders? Clin. Psychol. Sci. 2, 119–137. doi: 10.1177/2167702613497473

PubMed Abstract | CrossRef Full Text | Google Scholar

Cattell, R. B. (1966). “Patterns of change: measurement in relation to state-dimension, trait change, lability, and process concepts,” in Handbook of Multivariate Experimental Psychology, ed. R. B. Cattell (Chicago, IL: Rand McNally and Company), 355–402.

Google Scholar

Cervone, D. (2005). Personality architecture: within-person structures and processes. Annu. Rev. Psychol. 56, 423–452. doi: 10.1146/annurev.psych.56.091103.070133

PubMed Abstract | CrossRef Full Text | Google Scholar

Cohen, S., Kessler, R. C., and Gordon, L. (1995). “Strategies for measuring stress in studies of psychiatric and physical disorders,” in Measuring Stress: A Guide for Health and Social Scientists, eds S. Cohen, R. C. Kessler, and L. Gordon (New York, NY: Oxford University Press), 3–26.

Google Scholar

Cramer, A. O., Waldorp, L. J., van der Maas, H. L., and Borsboom, D. (2010). Comorbidity: a network perspective. Behav. Brain Sci. 33, 137–150. doi: 10.1017/S0140525X09991567

PubMed Abstract | CrossRef Full Text | Google Scholar

Cronbach, L. J., and Meehl, P. E. (1955). Construct validity in psychological tests. Psychol. Bull. 52, 281–302. doi: 10.1037/h0040957

CrossRef Full Text | Google Scholar

Cuthbert, B. N., and Insel, T. R. (2013). Toward the future of psychiatric diagnosis: the seven pillars of RDoC. BMC Med. 11:126. doi: 10.1186/1741-7015-11-126

PubMed Abstract | CrossRef Full Text | Google Scholar

DeYoung, C. G. (2015). Cybernetic big five theory. J. Res. Pers. 56, 33–58. doi: 10.1016/j.jrp.2014.07.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Digman, J. M. (1990). Personality structure: emergence of the five-factor model. Annu. Rev. Psychol. 41, 417–440. doi: 10.1146/annurev.ps.41.020190.002221

CrossRef Full Text | Google Scholar

Digman, J. M. (1996). “The curious history of the five-factor model,” in Theoretical Foundations of the Five-Factor Model, ed. J. S. Wiggins (New York, NY: Guilford Press).

Google Scholar

Eaton, N. R., Keyes, K. M., Krueger, R. F., Balsis, S., Skodol, A. E., Markon, K. E., et al. (2012). An invariant dimensional liability model of gender differences in mental disorder prevalence: evidence from a national sample. J. Abnorm. Psychol. 121, 282–288. doi: 10.1037/a0024780

PubMed Abstract | CrossRef Full Text | Google Scholar

Eaton, N. R., Krueger, R. F., and Oltmanns, T. F. (2011). Aging and the structure and long-term stability of the internalizing spectrum of personality and psychopathology. Psychol. Aging 26, 987–993. doi: 10.1037/a0024406

PubMed Abstract | CrossRef Full Text | Google Scholar

Ebner-Priemer, U. W., Kuo, J., Kleindienst, N., Welch, S. S., Reisch, T., Reinhard, I., et al. (2007). State affective instability in borderline personality disorder assessed by ambulatory monitoring. Psychol. Med. 37, 961–970. doi: 10.1017/S0033291706009706

PubMed Abstract | CrossRef Full Text | Google Scholar

Eid, M., and Diener, E. (1999). Intraindividual variability in affect: reliability, validity, and personality correlates. J. Pers. Soc. Psychol. 76, 662–676. doi: 10.1037/0022-3514.76.4.662

CrossRef Full Text | Google Scholar

Finn, S. E. (2007). In our Clients’ Shoes: Theory and Techniques of Therapeutic Assessment. Hove: Psychology Press.

Google Scholar

First, M. B., Gibbon, M., Spitzer, R. L., Williams, J. B. W., and Benjamin, L. S. (1997). Structured Clinical Interview for DSM-IV axis II Personality Disorders, (SCID-II). Washington, DC: American Psychiatric Press.

Google Scholar

Fleeson, W. (2001). Toward a structure-and process-integrated view of personality: traits as density distributions of states. J. Pers. Soc. Psychol. 80, 1011–1027. doi: 10.1037/0022-3514.80.6.1011

CrossRef Full Text | Google Scholar

Fleeson, W. (2007). Situation-based contingencies underlying trait-content manifestation in behavior. J. Pers. 75, 825–862. doi: 10.1111/j.1467-6494.2007.00458.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Fleeson, W., and Gallagher, P. (2009). The implications of Big Five standing for the distribution of trait manifestation in behavior: fifteen experience-sampling studies and a meta-analysis. J. Pers. Soc. Psychol. 97, 1097–1114. doi: 10.1037/a0016786

PubMed Abstract | CrossRef Full Text | Google Scholar

Fleeson, W., and Jayawickreme, E. (2015). Whole trait theory. J. Res. Pers. 56, 82–92. doi: 10.1016/j.jrp.2014.10.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Forbes, E. E., Stepp, S. D., Dahl, R. E., Ryan, N. D., Whalen, D., Axelson, D. A., et al. (2012). Real-world affect and social context as predictors of treatment response in child and adolescent depression and anxiety: an ecological momentary assessment study. J. Child Adolesc. Psychopharmacol. 22, 37–47. doi: 10.1089/cap.2011.0085

PubMed Abstract | CrossRef Full Text | Google Scholar

Forbush, K. T., and Watson, D. (2013). The structure of common and uncommon mental disorders. Psychol. Med. 43, 97–108. doi: 10.1017/S0033291712001092

PubMed Abstract | CrossRef Full Text | Google Scholar

Fournier, M. A., Moskowitz, D. S., and Zuroff, D. C. (2009). The interpersonal signature. J. Res. Pers. 43, 155–162. doi: 10.1016/j.jrp.2009.01.023

CrossRef Full Text | Google Scholar

Fried, E. I. (2015). Problematic assumptions have slowed down depression research: why symptoms, not syndromes are the way forward. Front. Psychol. 6:309. doi: 10.3389/fpsyg.2015.00309

PubMed Abstract | CrossRef Full Text | Google Scholar

Gates, K. M., and Molenaar, P. C. M. (2012). Group search algorithm recovers effective connectivity maps for individuals in homogeneous and heterogeneous samples. Neuroimage 63, 310–319. doi: 10.1016/j.neuroimage.2012.06.026

PubMed Abstract | CrossRef Full Text | Google Scholar

Gates, K. M., Molenaar, P. C., Hillary, F. G., Ram, N., and Rovine, M. J. (2010). Automatic search for fMRI connectivity mapping: an alternative to Granger causality testing using formal equivalences among SEM path modeling, VAR, and unified SEM. Neuroimage 50, 1118–1125. doi: 10.1016/j.neuroimage.2009.12.117

PubMed Abstract | CrossRef Full Text | Google Scholar

Gates, K. M., Molenaar, P. C. M., Hillary, F. G., and Slobounov, S. (2011). Extended unified SEM approach for modeling event-related fMRI data. Neuroimage 54, 1151–1158. doi: 10.1016/j.neuroimage.2010.08.051

PubMed Abstract | CrossRef Full Text | Google Scholar

Geldhof, G. J., Preacher, K. J., and Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychol. Methods 19, 72–91. doi: 10.1037/a0032138

PubMed Abstract | CrossRef Full Text | Google Scholar

Goldberg, L. R. (1993). The structure of phenotypic personality traits. Am. Psychol. 48, 26–34. doi: 10.1037/0003-066X.48.12.1303

CrossRef Full Text | Google Scholar

Grice, J. W. (2004). Bridging the idiographic-nomothetic divide in ratings of self and others on the big five. J. Pers. 72, 203–241. doi: 10.1111/j.0022-3506.2004.00261.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Hamaker, E. L., Dolan, C. V., and Molenaar, P. C. M. (2005). Statistical modeling of the individual: rationale and application of multivariate time series analysis. Multivar. Behav. Res. 40, 207–233. doi: 10.1207/s15327906mbr4002_3

CrossRef Full Text | Google Scholar

Hamaker, E. L., Nesselroade, J. R., and Molenaar, P. C. (2007). The integrated trait–state model. J. Res. Pers. 41, 295–315. doi: 10.1016/j.jrp.2006.04.003

CrossRef Full Text | Google Scholar

Harkness, A. R., Reynolds, S. M., and Lilienfeld, S. O. (2014). A review of systems for psychology and psychiatry: adaptive systems, personality psychopathology five (PSY-5), and the DSM-5. J. Pers. Assess. 96, 121–139. doi: 10.1080/00223891.2013.823438

PubMed Abstract | CrossRef Full Text | Google Scholar

Heck, R. H. (1999). “Multilevel modeling with SEM,” in Introduction to Multilevel Modeling Techniques, eds S. L. Thomas and R. H. Heck (Mahwah, NJ: Lawrence Erlbaum Associates, Inc.), 89–127.

Google Scholar

Hillary, F. G., Medaglia, J. D., Gates, K., Molenaar, P. C., Slocomb, J., Peechatka, A., et al. (2011). Examining working memory task acquisition in a disrupted neural network. Brain 134, 1555–1570. doi: 10.1093/brain/awr043

PubMed Abstract | CrossRef Full Text | Google Scholar

Hopwood, C. J., Zimmermann, J., Pincus, A. L., and Krueger, R. F. (2015). Connecting personality structure and dynamics: towards a more evidence based and clinically useful diagnostic scheme. J. Personal. Disord. 29, 431–448. doi: 10.1521/pedi.2015.29.4.431

PubMed Abstract | CrossRef Full Text | Google Scholar

Houben, M., Van Den Noortgate, W., and Kuppens, P. (2015). The relation between short term emotion dynamics and psychological well-being: a meta-analysis. Psychol. Bull. 141, 901–930. doi: 10.1037/a0038822

PubMed Abstract | CrossRef Full Text | Google Scholar

Hu, L. T., and Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equ. Model. Multidiscip. J. 6, 1–55. doi: 10.1080/10705519909540118

CrossRef Full Text | Google Scholar

Hyman, S. E. (2010). The diagnosis of mental disorders: the problem of reification. Ann. Rev. Clin. Psychol. 6, 155–179. doi: 10.1146/annurev.clinpsy.3.022806.091532

CrossRef Full Text | Google Scholar

Insel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., et al. (2010). Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am. J. Psychiatry 167, 748–751. doi: 10.1176/appi.ajp.2010.09091379

PubMed Abstract | CrossRef Full Text | Google Scholar

Jahng, S., Wood, P. K., and Trull, T. J. (2008). Analysis of affective instability in ecological momentary assessment: indices using successive difference and group comparison via multilevel modeling. Psychol. Methods 13, 354–375. doi: 10.1037/a0014173

PubMed Abstract | CrossRef Full Text | Google Scholar

Jöreskog, K. G., and Sörbom, D. (1992). LISREL. Chicago, IL: Scientific Software International, Inc.

Google Scholar

Kendler, K. S., Aggen, S. H., Knudsen, G. P., Røysamb, E., Neale, M. C., and Reichborn-Kjennerud, T. (2011). The structure of genetic and environmental risk factors for syndromal and subsyndromal common DSM-IV Axis I and All Axis II Disorders. Am. J. Psychiatry 168, 29–39. doi: 10.1176/appi.ajp.2010.10030340

PubMed Abstract | CrossRef Full Text | Google Scholar

Kessler, R. C., Chiu, W. T., Demler, O., and Walters, E. E. (2005). Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch. Gen. Psychiatry 62, 617–627. doi: 10.1001/archpsyc.62.6.617

PubMed Abstract | CrossRef Full Text | Google Scholar

Kessler, R. C., McGonagle, K. A., Zhao, S., Nelson, C. B., Hughes, M., Eshleman, S., et al. (1994). Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States: results from the National Comorbidity Survey. Arch. Gen. Psychiatry 51, 8–19. doi: 10.1001/archpsyc.1994.03950010008002

CrossRef Full Text | Google Scholar

Kim, J., Zhu, W., Chang, L., Bentler, P. M., and Ernst, T. (2007). Unified structural equation modeling approach for the analysis of multisubject, multivariate functional MRI data. Hum. Brain Mapp. 28, 85–93. doi: 10.1002/hbm.20259

PubMed Abstract | CrossRef Full Text | Google Scholar

Kotov, R., Chang, S. W., Fochtmann, L. J., Mojtabai, R., Carlson, G. A., Sedler, M. A., et al. (2010a). Schizophrenia in the internalizing–externalizing framework: a third-dimension? Schizophr. Bull. 37, 1168–1178. doi: 10.1093/schbul/sbq024

PubMed Abstract | CrossRef Full Text | Google Scholar

Kotov, R., Gamez, W., Schmidt, F., and Watson, D. (2010b). Linking “big” personality traits to anxiety, depressive, and substance use disorders: a meta-analysis. Psychol. Bull. 136, 768–821. doi: 10.1037/a0020327

PubMed Abstract | CrossRef Full Text | Google Scholar

Kotov, R., Ruggero, C. J., Krueger, R. F., Watson, D., Yuan, Q., and Zimmerman, M. (2011). New dimensions in the quantitative classification of mental illness. Arch. Gen. Psychiatry 68, 1003. doi: 10.1001/archgenpsychiatry.2011.107

PubMed Abstract | CrossRef Full Text | Google Scholar

Krueger, R. F. (1999). The structure of common mental disorders. Arch. Gen. Psychiatry 56, 921. doi: 10.1001/archpsyc.56.10.921

CrossRef Full Text | Google Scholar

Krueger, R. F., Caspi, A., Moffitt, T. E., and Silva, P. A. (1998). The structure and stability of common mental disorders (DSM-III-R): a longitudinal-epidemiological study. J. Abnorm. Psychol. 107, 216–227. doi: 10.1037/0021-843X.107.2.216

PubMed Abstract | CrossRef Full Text | Google Scholar

Krueger, R. F., and Markon, K. E. (2006). Reinterpreting comorbidity: a model-based approach to understanding and classifying psychopathology. Annu. Rev. Clin. Psychol. 2, 111–133. doi: 10.1146/annurev.clinpsy.2.022305.095213

CrossRef Full Text | Google Scholar

Krueger, R. F., Markon, K. E., Patrick, C. J., Benning, S. D., and Kramer, M. D. (2007). Linking antisocial behavior, substance use, and personality: an integrative quantitative model of the adult externalizing spectrum. J. Abnorm. Psychol. 116, 645–666. doi: 10.1037/0021-843X.116.4.645

PubMed Abstract | CrossRef Full Text | Google Scholar

Kushner, S. C., Quilty, L. C., Tackett, J. L., and Bagby, R. M. (2011). The hierarchical structure of the Dimensional Assessment of Personality Pathology (DAPP-BQ). J. Personal. Disord. 25, 504–516. doi: 10.1521/pedi.2011.25.4.504

PubMed Abstract | CrossRef Full Text | Google Scholar

Lahey, B. B., Rathouz, P. J., Van Hulle, C., Urbano, R. C., Krueger, R. F., Applegate, B., et al. (2008). Testing structural models of DSM-IV symptoms of common forms of child and adolescent psychopathology. J. Abnorm. Child Psychol. 36, 187–206. doi: 10.1007/s10802-007-9169-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Larsen, R. J. (1987). The stability of mood variability: a spectral analytic approach to daily mood assessments. J. Pers. Soc. Psychol. 52, 1195. doi: 10.1037/0022-3514.52.6.1195

CrossRef Full Text | Google Scholar

Linehan, M. M. (1993). Cognitive-Behavioral Treatment of Borderline Personality Disorder. New York, NY: Guilford Press.

Google Scholar

Livesley, W. J., Jang, K. L., and Vernon, P. A. (1998). Phenotypic and genetic structure of traits delineating personality disorder. Arch. Gen. Psychiatry 55, 941–948. doi: 10.1001/archpsyc.55.10.941

CrossRef Full Text | Google Scholar

MacCallum, R. C., Wegener, D. T., Uchino, B. N., and Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychol. Bull. 114, 185–199. doi: 10.1037/0033-2909.114.1.185

CrossRef Full Text | Google Scholar

Markon, K. E. (2010). Modeling psychopathology structure: a symptom-level analysis of Axis I and II disorders. Psychol. Med. 40, 273–288. doi: 10.1017/S0033291709990183

PubMed Abstract | CrossRef Full Text | Google Scholar

Markon, K. E., Krueger, R. F., and Watson, D. (2005). Delineating the structure of normal and abnormal personality: an integrative hierarchical approach. J. Pers. Soc. Psychol. 88, 139–157. doi: 10.1037/0022-3514.88.1.139

PubMed Abstract | CrossRef Full Text | Google Scholar

McCabe, K. O., and Fleeson, W. (2012). What is extraversion for? Integrating trait and motivational perspectives and identifying the purpose of extraversion. Psychol. Sci. 23, 1498–1505. doi: 10.1177/0956797612444904

PubMed Abstract | CrossRef Full Text | Google Scholar

Mischel, W., and Shoda, Y. (1995). A cognitive-affective system theory of personality: reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychol. Rev. 102, 246–268. doi: 10.1037/0033-295X.102.2.246

PubMed Abstract | CrossRef Full Text | Google Scholar

Molenaar, P. C. M. (1985). A dynamic factor model for the analysis of multivariate time series. Psychometrika 50, 181–202. doi: 10.1007/BF02294246

CrossRef Full Text | Google Scholar

Molenaar, P. C. (2004). A manifesto on psychology as idiographic science: bringing the person back into scientific psychology, this time forever. Measurement 2, 201–218.

Google Scholar

Molenaar, P. C., and Campbell, C. G. (2009). The new person-specific paradigm in psychology. Curr. Dir. Psychol. Sci. 18, 112–117. doi: 10.1111/j.1467-8721.2009.01619.x

CrossRef Full Text | Google Scholar

Moskowitz, D. S., and Zuroff, D. C. (2004). Flux, pulse, and spin: dynamic additions to the personality lexicon. J. Pers. Soc. Psychol. 86, 880–893. doi: 10.1037/0022-3514.86.6.880

PubMed Abstract | CrossRef Full Text | Google Scholar

Muthén, B. O. (1991). Multilevel factor analysis of class and student achievement components. J. Educ. Measure. 28, 338–354. doi: 10.1111/j.1745-3984.1991.tb00363.x

CrossRef Full Text | Google Scholar

Muthén, B. O. (1994). Multilevel covariance structure analysis. Sociol. Methods Res. 22, 376–398. doi: 10.1177/0049124194022003006

CrossRef Full Text | Google Scholar

Muthén, L. K., and Muthén, B. O. (1998–2012). Mplus User’s Guide, 7th Edn. Los Angeles, CA: Author.

Myin-Germeys, I., Oorschot, M., Collip, D., Lataster, J., Delespaul, P., and van Os, J. (2009). Experience sampling research in psychopathology: opening the black box of daily life. Psychol. Med. 39, 1533–1547. doi: 10.1017/S0033291708004947

PubMed Abstract | CrossRef Full Text | Google Scholar

Nesselroade, J. R., Gerstorf, D., Hardy, S. A., and Ram, N. (2007). Focus article: idiographic filters for psychological constructs. Measurement 5, 217–235.

Google Scholar

Nesselroade, J. R., and Molenaar, P. C. M. (1999). “Pooling lagged covariance structures based on short, multivariate time series for dynamic factor analysis,” in Statistical Strategies for Small Sample Research, ed. R. Holye (New burry Park, CA: Sage), 223–251.

Google Scholar

Nolen-Hoeksema, S. (1991). Responses to depression and their effects on the duration of depressive episodes. J. Abnorm. Psychol. 100, 569–582. doi: 10.1037/0021-843X.100.4.569

CrossRef Full Text | Google Scholar

Pe, M. L., Kircanski, K., Thompson, R. J., Bringmann, L. F., Tuerlinckx, F., Mestdagh, M., et al. (2015). Emotion-network density in major depressive disorder. Clin. Psychol. Sci. 3, 292–300. doi: 10.1177/2167702614540645

CrossRef Full Text | Google Scholar

Preacher, K. J., Zyphur, M. J., and Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychol. Methods 15, 209–233. doi: 10.1037/a0020141

PubMed Abstract | CrossRef Full Text | Google Scholar

Read, S. J., Monroe, B. M., Brownstein, A. L., Yang, Y., Chopra, G., and Miller, L. C. (2010). A neural network model of the structure and dynamics of human personality. Psychol. Rev. 117, 61–92. doi: 10.1037/a0018131

PubMed Abstract | CrossRef Full Text | Google Scholar

Reise, S. P., Ventura, J., Nuechterlein, K. H., and Kim, K. H. (2005). An illustration of multilevel factor analysis. J. Pers. Assess. 84, 126–136. doi: 10.1207/s15327752jpa8402_02

PubMed Abstract | CrossRef Full Text | Google Scholar

Revelle, W., and Condon, D. M. (2015). A model for personality at three levels. J. Res. Pers. 56, 70–81. doi: 10.1016/j.jrp.2014.12.006

CrossRef Full Text | Google Scholar

Røysamb, E., Kendler, K. S., Tambs, K., Ørstavik, R. E., Neale, M. C., Aggen, S. H., et al. (2011). The joint structure of DSM-IV Axis I and Axis II disorders. J. Abnorm. Psychol. 120, 198–209. doi: 10.1037/a0021660

PubMed Abstract | CrossRef Full Text | Google Scholar

Ruiz, M. A., Pincus, A. L., and Schinka, J. A. (2008). Externalizing pathology and the five-factor model: a meta-analysis of personality traits associated with antisocial personality disorder, substance use disorder, and their co-occurrence. J. Personal. Disord. 22, 365–388. doi: 10.1521/pedi.2008.22.4.365

PubMed Abstract | CrossRef Full Text | Google Scholar

Russell, J. J., Moskowitz, D. S., Zuroff, D. C., Sookman, D., and Paris, J. (2007). Stability and variability of affective experience and interpersonal behavior in borderline personality disorder. J. Abnorm. Psychol. 116, 578–588. doi: 10.1037/0021-843X.116.3.578

PubMed Abstract | CrossRef Full Text | Google Scholar

Ryu, E., and West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Struct. Equ. Model. 16, 583–601. doi: 10.1080/10705510903203466

CrossRef Full Text | Google Scholar

Sadikaj, G., Moskowitz, D. S., Russell, J. J., Zuroff, D. C., and Paris, J. (2013). Quarrelsome behavior in borderline personality disorder: influence of behavioral and affective reactivity to perceptions of others. J. Abnorm. Psychol. 122, 195–207. doi: 10.1037/a0030871

PubMed Abstract | CrossRef Full Text | Google Scholar

Samuel, D. B., and Widiger, T. A. (2008). A meta-analytic review of the relationships between the five-factor model and DSM-IV-TR personality disorders: a facet level analysis. Clin. Psychol. Rev. 28, 1326. doi: 10.1016/j.cpr.2008.07.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Satorra, A., and Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika 75, 243–248. doi: 10.1007/s11336-009-9135-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Saulsman, L. M., and Page, A. C. (2004). The five-factor model and personality disorder empirical literature: a meta-analytic review. Clin. Psychol. Rev. 23, 1055–1085. doi: 10.1016/j.cpr.2002.09.001

CrossRef Full Text | Google Scholar

Sheehan, D. V., and Lecrubier, Y. (2010). The Mini International Neuropsychiatric Interview, Version 6.0 (MINI 6.0). Jacksonville, FL: Medical Outcomes Systems, Inc.

Google Scholar

Shiffman, S., Gwaltney, C. J., Balabanis, M. H., Liu, K. S., Paty, J. A., Kassel, J. D., et al. (2002). Immediate antecedents of cigarette smoking: an analysis from ecological momentary assessment. J. Abnorm. Psychol. 111, 531–545. doi: 10.1037/0021-843X.111.4.531

CrossRef Full Text | Google Scholar

Shumway, R. H., and Stoffer, D. S. (2006). Time Series Analysis and its Applications: With R Examples, 2nd Edn. New York, NY: Springer.

Google Scholar

Silk, J. S., Steinberg, L., and Morris, A. S. (2003). Adolescents’ emotion regulation in daily life: links to depressive symptoms and problem behavior. Child Dev. 74, 1869–1880. doi: 10.1046/j.1467-8624.2003.00643.x

CrossRef Full Text | Google Scholar

Simms, L. J., Goldberg, L. R., Roberts, J. E., Watson, D., Welte, J., and Rotterman, J. H. (2011). Computerized adaptive assessment of personality disorder: introducing the CAT–PD Project. J. Pers. Assess. 93, 380–389. doi: 10.1080/00223891.2011.577475

PubMed Abstract | CrossRef Full Text | Google Scholar

Slade, T., and Watson, D. (2006). The structure of common DSM-IV and ICD-10 mental disorders in the Australian general population. Psychol. Med. 36, 1593–1600. doi: 10.1017/S0033291706008452

PubMed Abstract | CrossRef Full Text | Google Scholar

Sörbom, D. (1989). Model modification. Psychometrika 54, 371–384. doi: 10.1007/BF02294623

CrossRef Full Text | Google Scholar

Stepp, S. D., Hallquist, M. N., Morse, J. Q., and Pilkonis, P. A. (2011). Multimethod investigation of interpersonal functioning in borderline personality disorder. Pers. Disord. Theor. Res. Treat. 2, 175–192. doi: 10.1037/a0020572

PubMed Abstract | CrossRef Full Text | Google Scholar

Teasdale, J. D. (1988). Cognitive vulnerability to persistent depression. Cogn. Emot. 2, 247–274. doi: 10.1080/02699938808410927

CrossRef Full Text | Google Scholar

Timmerman, M. E. (2006). Multilevel component analysis. Br. J. Mathemat. Statist. Psychol. 59, 301–320. doi: 10.1348/000711005X67599

PubMed Abstract | CrossRef Full Text | Google Scholar

Titchener, E. B. (1898). The postulates of a structural psychology. Philos. Rev. 7, 449–465. doi: 10.2307/2177110

CrossRef Full Text | Google Scholar

Tomko, R. L., Solhan, M. B., Carpenter, R. W., Brown, W. C., Jahng, S., Wood, P. K., et al. (2014). Measuring impulsivity in daily life: the momentary impulsivity scale. Psychol. Assess. 26, 339–349. doi: 10.1037/a0035083

PubMed Abstract | CrossRef Full Text | Google Scholar

Trull, T. J., Solhan, M. B., Tragesser, S. L., Jahng, S., Wood, P. K., Piasecki, T. M., et al. (2008). Affective instability: measuring a core feature of borderline personality disorder with ecological momentary assessment. J. Abnorm. Psychol. 117, 647–661. doi: 10.1037/a0012532

PubMed Abstract | CrossRef Full Text | Google Scholar

Vollebergh, W. A., Iedema, J., Bijl, R. V., de Graaf, R., Smit, F., and Ormel, J. (2001). The structure and stability of common mental disorders: the NEMESIS study. Arch. Gen. Psychiatry 58, 597–603. doi: 10.1001/archpsyc.58.6.597

PubMed Abstract | CrossRef Full Text | Google Scholar

von Eye, A., and Bergman, L. R. (2003). Research strategies in developmental psychopathology: dimensional identity and the person-oriented approach. Dev. Psychopathol. 15, 553–580. doi: 10.1017/S0954579403000294

PubMed Abstract | CrossRef Full Text | Google Scholar

Wegner, K. E., Smyth, J. M., Crosby, R. D., Wittrock, D., Wonderlich, S. A., and Mitchell, J. E. (2002). An evaluation of the relationship between mood and binge eating in the natural environment using ecological momentary assessment. Int. J. Eat. Disord. 32, 352–361. doi: 10.1002/eat.10086

PubMed Abstract | CrossRef Full Text | Google Scholar

Wichers, M. (2014). The dynamic nature of depression: a new micro-level perspective of mental disorder that meets current challenges. Psychol. Med. 44, 1349–1360. doi: 10.1017/S0033291713001979

PubMed Abstract | CrossRef Full Text | Google Scholar

Widiger, T. A., and Simonsen, E. (2005). Alternative dimensional models of personality disorder: finding a common ground. J. Personal. Disord. 19, 110–130. doi: 10.1521/pedi.19.2.110.62628

PubMed Abstract | CrossRef Full Text | Google Scholar

Widiger, T. A., and Trull, T. J. (2007). Plate tectonics in the classification of personality disorder: shifting to a dimensional model. Am. Psychol. 62, 71–83. doi: 10.1037/0003-066X.62.2.71

PubMed Abstract | CrossRef Full Text | Google Scholar

Wigman, J. T. W., van Os, J., Borsboom, D., Wardenaar, K. J., Epskamp, S., Klippel, A., et al. (2015). Exploring the underlying structure of mental disorders: cross-diagnostic differences and similarities from a network perspective using both a top-down and a bottom-up approach. Psychol. Med. 45, 2375–2387. doi: 10.1017/S0033291715000331

PubMed Abstract | CrossRef Full Text | Google Scholar

Wolf, A. W., Schubert, D. S. P., Patterson, M. B., Grande, T. P., Brocco, K. J., and Pendleton, L. (1988). Associations among major psychiatric diagnoses. J. Consult. Clin. Psychol. 56, 292–294. doi: 10.1037/0022-006X.56.2.292

CrossRef Full Text | Google Scholar

Wright, A. G. C. (in press). “Factor analytic support for the five-factor model,” in Oxford Handbook of the Five-Factor Model, ed. T. A. Widiger (Oxford: Oxford University Press). doi: 10.1093/oxfordhb/9780199352487.013.20

CrossRef Full Text

Wright, A. G. C. (2011). Quantitative and qualitative distinctions in personality disorder. J. Pers. Assess. 93, 370–379. doi: 10.1080/00223891.2011.577477

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C. (2014). “Integrating trait and process based conceptualizations of pathological narcissism in the DSM-5 era,” in Handbook of Psychology of Narcissism: Diverse Perspectives, ed. A. Besser (Hauppauge, NY: Nova Science Publishers), 153–174.

Google Scholar

Wright, A. G., Hallquist, M. N., Swartz, H. A., Frank, E., and Cyranowski, J. M. (2014). Treating co-occurring depression and anxiety: modeling the dynamics of psychopathology and psychotherapy using the time-varying effect model. J. Consult. Clin. Psychol. 82, 839–853. doi: 10.1037/a0034430

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C., Hopwood, C. J., and Simms, L. J. (2015). Daily interpersonal and affective dynamics in personality disorder. J. Personal. Disord. 29, 503–525. doi: 10.1521/pedi.2015.29.4.503

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C., Krueger, R. F., Hobbs, M. J., Markon, K. E., Eaton, N. R., and Slade, T. (2013). The structure of psychopathology: toward an expanded quantitative empirical model. J. Abnorm. Psychol. 122, 281–294. doi: 10.1037/a0030133

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C., and Simms, L. J. (2014). On the structure of personality disorder traits: conjoint analyses of the CAT-PD, PID-5, and NEO-PI-3 trait models. Pers. Disord. Theor. Res. Treat. 5, 43–54. doi: 10.1037/per0000037

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C., and Simms, L. J. (2015). A metastructural model of mental disorders and pathological personality traits. Psychol. Med. 45, 2309–2319. doi: 10.1017/S0033291715000252

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C., Thomas, K. M., Hopwood, C. J., Markon, K. E., Pincus, A. L., and Krueger, R. F. (2012). The hierarchical structure of DSM-5 pathological personality traits. J. Abnorm. Psychol. 121, 951–957. doi: 10.1037/a0027669

PubMed Abstract | CrossRef Full Text | Google Scholar

Wright, A. G. C., and Zimmermann, J. (2015). “At the nexus of science and practice: answering basic clinical questions in personality disorder assessment and diagnosis with quantitative modeling techniques,” in Personality Disorders: Toward Theoretical and Empirical Integration in Diagnosis and Assessment, ed. S. Huprich (Washington, DC: American Psychological Association), 109–144.

Google Scholar

Zimmerman, M., and Mattia, J. I. (1999). Psychiatric diagnosis in clinical practice: is comorbidity being missed? Compr. Psychiatry 40, 182–191. doi: 10.1016/S0010-440X(99)90001-9

CrossRef Full Text | Google Scholar