Eleni A. Demetriou1†
Shin H. Park1†
Nicholas Ho1Karen L. Pepper1Yun J. C. Song1
Sharon L. Naismith2Emma E. Thomas1
Ian B. Hickie1,3
Adam J. Guastella1,3*- 1Autism Clinic for Translational Research, Child Neurodevelopment and Mental Health Team, Brain and Mind Centre, Children’s Hospital Westmead l Clinical School, Faculty of Medicine and Health, University of Sydney, Sydney, NSW, Autstralia
- 2School of Psychology, University of Sydney, Sydney, NSW, Autstralia
- 3Youth Mental Health Unit, Brain and Mind Centre, Central Clinical School, Faculty of Medicine and Health, University of Sydney, Sydney, NSW, Autstralia
Differential diagnosis in adult cohorts with social difficulty is confounded by comorbid mental health conditions, common etiologies, and shared phenotypes. Identifying shared and discriminating profiles can facilitate intervention and remediation strategies. The objective of the study was to identify salient features of a composite test battery of cognitive and mood measures using a machine learning paradigm in clinical cohorts with social interaction difficulties. We recruited clinical participants who met standardized diagnostic criteria for autism spectrum disorder (ASD: n = 62), early psychosis (EP: n = 48), or social anxiety disorder (SAD: N = 83) and compared them with a neurotypical comparison group (TYP: N = 43). Using five machine-learning algorithms and repeated cross-validation, we trained and tested classification models using measures of cognitive and executive function, lower- and higher-order social cognition and mood severity. Performance metrics were the area under the curve (AUC) and Brier Scores. Sixteen features successfully differentiated between the groups. The control versus social impairment cohorts (ASD, EP, SAD) were differentiated by social cognition, visuospatial memory and mood measures. Importantly, a distinct profile cluster drawn from social cognition, visual learning, executive function and mood, distinguished the neurodevelopmental cohort (EP and ASD) from the SAD group. The mean AUC range was between 0.891 and 0.916 for social impairment versus control cohorts and, 0.729 to 0.781 for SAD vs neurodevelopmental cohorts. This is the first study that compares an extensive battery of neuropsychological and self-report measures using a machine learning protocol in clinical and neurodevelopmental cohorts characterized by social impairment. Findings are relevant for diagnostic, intervention and remediation strategies for these groups.
Introduction
Machine learning (ML) paradigms have facilitated the evaluation of complex datasets (1, 2) and provide a dynamic framework to enhance comparisons between groups that may share neurodevelopmental, clinical or cognitive profiles (3). In contrast to the traditional multiple regression methods, the ML algorithms are also capable of including many input variables with relatively smaller sample sizes (4, 5) and can handle both linear and non-linear interactions between variables. In medicine and psychology, the resultant algorithms have led to insights in clinical classification within (6) and between clinical cohorts (7), transdiagnostic subtyping of mental health symptoms (8) and comparative lifetime health outcomes (9). Such research may contribute to improved profiling of cohorts such as Schizophrenia (SCH) and Autism Spectrum Disorder (ASD) given shared genetic liability (10) and theorized common etiologies associated with social cognition (11) and executive function (EF) (12) processes.
The clinical sub-groups of SCH and ASD have drawn much debate about similarities and differences that might exist between the two diagnoses (13, 14). There is considerable empirical support of shared genetic, neurocognitive, and behavioral pathways between SCH and ASD (15–17). In both, co-morbidities appear higher than expected population outcomes (18) and impairments in cognitive function appear similarly in domains of social cognition (19) and EF (20). For ASD, diagnosis may be made as early as 18 months of age, however a proportion is diagnosed in adolescence/adulthood (21). The developmental course of psychosis is different, with a slow progression beginning with social withdrawal and early psychosis (EP) (22) that typically begins in later adolescence and early adulthood. In these cases, a third of people who develop EP will go on to develop SCH (23). There has been limited research exploring cognitive markers that may assist differential diagnosis. Such comparisons are particularly useful in early adulthood prior to the chronic manifestation of SCH symptoms to permit early differentiation of these disorders.
In this study we adopted the cognitive domains framework outlined in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (24). The DSM-5 (24) defines six cognitive domains as key domains for the assessment of neurocognitive disorders. These are complex attention, EF, learning and memory, language, perceptual–motor function, and social cognition. Within this framework complex attention refers to the processes of sustained attention (capacity to maintain attention on a discrete task over prolonged period), divided attention (focusing on two tasks simultaneously), selective attention (focusing on a specific task and ignoring others) and information processing speed. While acknowledging the considerable debate on the conceptualization of executive function (EF) (25, 26), research presented in this paper adopts the fractionated view of EF as noted in the DSM-5. Specifically, EF is characterized by discrete domains representing higher order cognitive processes. The EF domains include mental/cognitive flexibility (ability to shift between concepts), inhibition (ability to inhibit a previously learned or prepotent response), planning (ability to execute a sequence of actions so that a desired goal is achieved) and working memory (ability to store and dynamically manipulate information in temporary STM) (27).
In studies of complex attention, impairment in sustained attention has been reported in ASD (28) and on a composite battery of attention measures in EP (29). A recent comparison between EP and ASD (20) showed the former was significantly more impaired on attentional processes. Complex or top/down attentional processes have been shown to be guided by frontal neural circuitry (30), impairment noted above may reflect atypical processing in the prefrontal cortex in the ASD and EP groups, respectively.
Empirical findings in cognitive domains—other than EF and social cognition—are mixed and, in part dependent on the modality studied (verbal versus visuospatial). Studies with participants diagnosed with ASD have reported impaired performance in verbal learning (31), visuospatial short term memory (STM) (28), whilst others, noted superior visual (32) and comparable verbal STM (31, 32). In populations with psychosis, verbal STM and learning have also been noted to be impaired (12, 33, 34) but there have been mixed results for visual learning (33, 34). A study examining language domain measures in ASD with a neurotypical comparison group (35) found no differences between them. For the perceptual-motor domain difficulties have been reported for EP (34) and ASD (36).
There is, a larger body of research examining social cognition and EF domains and their contribution to symptoms and disability in each of the ASD and EP cohorts. Lower- and higher-order social cognition (37) performance has been shown to be reduced in participants diagnosed with either EP or ASD. These include performance on tests of emotion recognition (38, 39) and theory of mind tasks (34, 40, 41). Reduced performance on neurocognition has been reported for EP (22, 42) and ASD (36, 43, 44). Specifically, in relation to EF, impairment in EP has been reported in attentional shifting (20) with mixed findings across other domains including working memory and abstract thinking (33, 45). A recent meta-analysis in ASD across six EF domains (44) points to broad executive problems, likely characterized by aberrant neural network connectivity (46).
While there is evidence of difficulties across some of these cognitive domains for EP/SCH and ASD, respectively, few studies have directly compared EP with ASD. There is however, a greater body of literature comparing ASD and SCH. The overall findings on shared and distinct pathways remain equivocal and are to some degree moderated by the type of assessment used (e.g. behavioral, imaging, physiological). Greater commonalities are observed when comparing ASD and SCH on behavioral measures of social cognition and EF. Using an extensive battery of social cognition and EF tasks (47) comparable impairment was reported between the two clinical groups that was significantly worse than the neurotypical control group. Similar findings were reported in a recent study (48) on a battery of social cognition tests across the domains of emotion recognition, social perception, mental state attribution, and attributional style. The comparison between a group with ASD and a mixed cohort with SCH or schizoaffective disorder revealed comparable levels of impairment. It was further noted that the few significant differences between groups were mediated by symptom severity. Findings of comparable performance on behavioral measures of social cognition tasks however are mixed. Highlighting the importance of using stimuli with greater ecological validity, comparable performance was observed between ASD and SCH on emotion recognition tasks when stimuli were presented within a realistic contextual background. Furthermore, both groups were impaired compared to the neurotypical control group (49). Despite these similarities it was noted that IQ was a significant moderator for SCH but not for the ASD group. Findings suggest different cognitive processes may mediate the observed outcomes. Further evidence of differences between the two conditions on a behavioral attribution style task were reported in a meta-analysis (50). Greater impairment was observed in the ASD cohort compared to SCH. It was further observed that the transition from first episode psychosis to SCH resulted in greater impairment in the use of mental states in the SCH group compared to the EP group.
More differences between ASD and SCH emerge when underlying neural mechanisms are investigated even when the two cohorts are comparable on behavioral task performance. Using a task of perspective taking, (51) comparable performance was reported between ASD and SCH on behavioral tasks. Imaging data however, revealed that the two groups were distinguished by different functional connectivity outcomes with greater local orbitofrontal connectivity in ASD compared to SCH. Similar discrepancy between behavioral and neural outcomes were reported in a study utilizing a social judgment task (52). Comparable performance on the behavioral measure was guided by distinct neural mechanisms in the amygdala and associated neural circuit clusters and differentiated between individuals with ASD and schizotypal personality disorder. In two studies Ciarramidaro and associates examined intention attribution (53) and facial affect recognition (54) comparing individuals with ASD and paranoid SCH. In the former study, different neural mechanisms were observed, verifying the hyper-hypo connectivity hypothesis (55), despite comparable difficulties in making attributions between the two groups. In contrast, the latter study did not identify significant neural or behavioral differences between the two groups on the task used (implicit negative affect recognition task).
These findings highlight that methodological design including diagnostic subtypes, transition stage of SCH (early or chronic psychosis), type of task and assessment mode, may contribute to the observed behavioral phenotype and underlying neural mechanisms. ML provides a methodology that allows for multiple variates to be assessed in a single design and can thus make a significant contribution to this field. Applying this methodology to a comparison between ASD and EP, prior to transition to psychosis and entrenchment of chronic symptomatology could provide significant insights on the neurodevelopmental basis of SCH and shared and distinct profiles between the two groups.
Furthermore, the ML methodology and focus on conditions with social impairment adopted in this study presents a novel approach for guiding research in this area. In particular, ML allows for the evaluation of multifactorial assessment outcomes, this is particularly important given potential biases in self/informant assessments (20, 56). Identifying discriminating behavioral profiles can provide a framework for investigating mechanisms underlying the reported shared and distinct phenotypes.
Few studies (19, 20) compared the EP and ASD groups with a clinical group that shares the social impairment phenotype but has generally intact cognitive and EF (57) such as Social Anxiety Disorder (SAD). The SAD group presents an important comparison cohort given that EP (58) and ASD (59) are both associated with substantially elevated levels of social anxiety reported at 25% and 50%, respectively. In addition, there is a period of prodromal features that are difficult to distinguish between SAD and early psychosis (60). A comparison between the three groups could facilitate discriminating profiles and aid diagnosis.
The broad goal of this study was to use ML on a large dataset of multiple cognitive domain measures and mood self-appraisals. The aims were to identify differentiating profiles between neurodevelopmental (EP, ASD), clinical (SAD) and neurotypical (TYP) comparison groups. Identifying discriminating profiles between these conditions would facilitate diagnosis and early intervention.
Our assessment battery included multiple measures across the domains of complex attention, executive function, learning and memory, perceptual-motor function, and social cognition. Self-report measures of depression, anxiety, and stress were also included in the study given research evidence demonstrating high levels of co-morbid depression in SAD (48%) (61) with reported range between 35% and 70% (62, 63). Comparable rates of depression comorbidities (54%) have been reported for EP (64). In adults with ASD, the rate of depression disorders range from 38% to 70%, while the rate of anxiety disorders ranges from 50% to 65% (65, 66). To our knowledge, this is the first study to compare EP and ASD with SAD and a non-clinical comparison group across broad cognitive domains and affective states.
The first aim was to identify a profile that may distinguish between the combined social impairment cohort and control group. The second aim was to identify variables that differentiated the neurodevelopmental cohort from the SAD group. The third aim was to determine whether each of the EP, ASD, and SAD groups could be distinguished on a subset of measures from the other clinical groups and from each other. We predicted that the neurotypical control group would be distinguished from the social impairment cohort on self-appraisal measures of depression and anxiety, given the reported high comorbidity rates in the clinical groups. Second, we predicted that the neurodevelopmental cohort would be distinguished from the clinical comparison group on measures of attention, psychomotor speed, social cognition, EF, and visuomotor performance. This is based on literature findings that these domains are generally intact in SAD (57) but impaired in ASD (28, 31) and EP (34). Third, we predicted that the EP and ASD groups would be distinguished from each other on measures of complex attention given empirical support for impaired neural circuitry underpinning attention networks in EP (67). In their review, Wood and associates showed that attentional switching predicted transition from EP to SCH. This may be a useful marker for differential diagnosis. No specific predictions were made for the comparisons of ASD versus SAD/EP and EP versus SAD/ASD.
Methods
Ethics
Ethics approval was given by the University of Sydney Ethics Committee (Protocol number 2013/352). Informed consent was obtained from each of the participants by postgraduate research students and trained clinicians.
Participants
Our dataset consisted of clinical participants who have met standardized diagnostic criteria for ASD (N = 62), EP (N = 48) or SAD (N = 83). Participants were sequential referrals from the Autism Clinic for Translational Research, Anxiety Clinic, and headspace clinics, at the Brain and Mind Centre, University of Sydney. Neurotypical control study volunteers (TYP = 43), were recruited separately through advertising at university websites. Clinical diagnoses were based on standardized diagnostic instruments (ADOS (68), ADIS-IV/V (69), SCID-I (70), PANS (71), and IQ was assessed based on scores on the WASI (72) or WTAR (73). Participants were excluded if IQ was below 70, prospective TYP participants were excluded if they reported past or current mental health diagnosis, or of they scored above cut-offs on screening instruments of depression, anxiety/social anxiety, stress or autism, [DASS-21 (74), SIAS (75), AQ-10] (76). Details of the diagnostic and assessment batteries and associated cognitive domains are presented in Table 1.
Table 1 Summary of assessment measures.
Assessment Battery
In this study, we utilized both neuropsychological (objective) and self-report (subjective) measures of social cognition, cognitive, and executive function, as well as self-report measures of affective states (depression, anxiety, and stress).
Data Selection
Patients and variables with more than 50% missing values were removed from the data set and the remaining missing data values were imputed with multivariate imputation with chained equations (MICE) (90) with 10 iterations using predictive mean matching for missing values. As shown in Figure 1, 10-fold cross-validation was applied to empirically assess the performance of the model built in imputed data sets. In each fold, 90% of the samples were used as the training set, and the remaining 10% were used for testing the generalizability of the models on unseen data. Ten-fold cross validation was used because it has been shown empirically to yield a reasonably low bias and modest variance (91, 92). Plausibility and consistency of the imputed values were visually inspected through density plots of the observed and imputed data, and the first imputed dataset was selected for downstream analysis.
Figure 1 Data selection flowchart.
Machine Learning
We applied five different ML algorithms to build models that can classify between our groups of interest, because there is no best algorithm for all problems (93). A great model for one problem may not hold for another problem. In particular, we selected five algorithms that can also perform variable selection in order to ascertain a variable's contribution to the model: Area Under the Curve Random Forests (AUCRF) (94), Boruta (95), Lasso regression (96), Elastic net regression (97) and Bayesian Additive Regression Trees (BART) (98).
The performances of the ML models were assessed using the Area Under the Curve (AUC) and Brier Scores. The AUC represents the probability that a classifier will ranks a randomly chosen negative example lower than a randomly chosen positive example (99). The AUC is a widely used performance measure in machine learning, and is often used as the primary performance measure for binary classification (100). In the evaluation of ML algorithms, the AUC has been shown to be a statistically consistent and more informative metric as compared to other traditionally used metrics, such as accuracy, precision, and recall (101–103). AUC is known to be a more complete performance metric as compared to other traditionally used metrics. AUC values < 0.5 suggest no discrimination, 0.7 to 0.8 are considered acceptable, 0.8 to 0.9 are considered good, and ≥ 0.9 are considered outstanding (104). As the AUC only represents the ability of a prediction model to distinguish between classes (discrimination), the Brier score was additionally used to evaluate the magnitude of the error of the probability estimates (calibration and discrimination) for complementing the AUC (105). Brier scores range between 0 (perfect accuracy) and 1 (perfect inaccuracy). Higher AUC and lower Brier scores indicate which model is the most informative. For those with similar scores, repeatedly identified features would be the reliable and clinically informative discriminatory features. Furthermore, to provide additional insights and make the results comparable to other studies that report accuracy, precision (positive predictive value), and recall (sensitivity) as performance evaluation measures, we have included these three measures in our analysis (refer to Table 3).
Machine Learning Algorithms
The AUCRF (94) and Boruta (106) algorithms are both based on the Random Forest (RF) (107) algorithm. RF uses bootstraps of samples to build a forest of decision trees with variables as nodes of the tree. Furthermore, RF has an internal variable importance ranking system that describes the decrease in node impurity. A higher-ranking variable is one that splits the samples into more pure groups.
AUCRF recursively builds RF models whilst eliminating the lowly ranked variables. The optimal set of variables is those used in the RF model with the best performance. For AUCRF, the metric for performance is the AUC which describes the model's true positive rate (sensitivity) and false positive rate (1-specificity) across different thresholds for binary classification.
Boruta uses RF to compare a variable's original importance score to its importance score from a permutation of that variable. Permutations break the relationship between the predictor and the response variables and, hence, are expected to decrease the predictive value of a variable. Variables with higher importance scores than in its permuted form are considered important. RF models were built with the optimal sets of variables as identified by AUCRF and Boruta, and tested to obtain performance metrics. For each RF model in AUCRF and Boruta and for each final RF model, we generated 1000 decision trees. The best variable for splitting at each level of each decision tree was identified from a random set of √p variables where p is the total number of variables. We also used internal 5-fold cross validation and a parameter tune length of 10 to identify the optimal value for λ (lambda). Lambda controls the strength of the penalization in Lasso and Elastic net and the balance between L1- and L2-regularization in Elastic net. Lasso (96) regression uses L1-regularization that penalizes coefficients with large absolute values in order to reduce overfitting. Lasso regression shrinks the coefficient of unimportant variables to zero and hence, effectively, performs variable selection. In contrast, Elastic net regression (97) employs a linear combination of L1-regularization and L2-regularization, which penalizes coefficients with large squared values. We used internal five-fold cross validation and a parameter tune length of 10 to identify the optimal value for λ which controls the strength of the penalization in Lasso and Elastic net, and α which controls the balance between L1- and L2-regularization in Elastic net.
In contrast to RF where trees are built from random bootstraps and independently, BART (98) employs a sum-of-trees approach. The Bayesian foundations of BART allows for the specification of regularization priors that ensures that each tree is weak and the use of Bayesian back-fitting (108) to fit trees iteratively. Variable selection with BART involves comparing the variable's inclusion proportions, which reflects the frequency of which the variable is chosen to be the split node, against a null distribution created from multiple permutations of the variable.
Results
Sample Description
Demographic characteristics are summarized in Table 2. In total, 236 participants were included in the study with mean age X = 22.7 years (SD 5.8), 96 (40.7%) were female. No significant differences were observed between the cohorts with regard to age, gender, and years of education (p > 0.05).
Table 2 Demographic descriptive statistics by diagnosis.
Model Performance
Classification performance for classifying between neurotypical controls and social impairment cohorts was good with mean AUCs greater than 0.87 (Table 3). All five algorithms performed similarly well with BART providing the highest mean AUC (0.92) and Boruta providing the lowest mean Brier Score (0.14).
Table 3 Classification performance on repeated cross-validation test sets.
For classification between clinical and neurodevelopmental groups, the mean AUCs were lower than that between neurotypical controls and the combined social impairment cohort, which reflects the challenge in developing a classification tool between disorders. Mean AUCs for discrimination between the EP and ASD groups ranged from 0.72 to 0.76 with BART providing the highest mean AUC (0.76) and Boruta the lowest mean Brier Score (0.21).
Variable Selection
Figure 2A illustrates the frequency that each variable was selected across the five algorithms with repeated cross-validation in the social impairment vs neurotypical control group. Self-report measures of depression, anxiety, and stress (DASS), social cognition measures of EQ-social skills, and the cognitive measure of visuospatial STM, best discriminated between the groups.
Figure 2 (A) Heatmap of Neurotypical Control vs Social Impairment Cohort. (B) Heatmap of SAD vs Neurodevelopmental Cohort. (E) Heatmap of ASD vs EP and SAD. (D) Heatmap of EP vs ASD and SAD. (E) Heatmap of ASD vs EP. (A–E) Variable frequency. Plot shows the frequency that each variable was selected for differentiating Control and Social impairment cohorts across the five algorithms with 10 times repeated 10-fold cross-validation. The darker color represents high frequency, while the lighter color represents low frequency.
Figures 2B–E and Table 4 present the top variables identified from the variables input into the three models for differentiating diagnosis between the disorders of social impairments by all five algorithms. Discriminating variables were identified across cognitive domains and affective states. A summary of the key discriminating variables is presented in Figure 3.
Table 4 Variables discriminating between ASD, EP, SAD, and neurotypical controls.
Figure 3 Venn diagram of distinguishing features.
Performance metrics like AUC and Brier scores represent how well the model differentiates between patients with different mental health problems. Based on the AUC and Brier scores, the most informative model can be identified. For example, the model for differentiating diagnosis between the SAD and neurodevelopmental group, the BART model showed the highest AUC = 0.781 and the lowest Brier score (0.198). However, other models also showed similar AUC and Brier scores. Features that are consistently selected by the best-performing model and these similarly performing models can be the recommendations for diagnosis and intervention strategies.
Discussion
In this study, we used ML algorithms on a composite assessment battery to identify cognitive profiles that discriminate between clinical, neurodevelopmental, and neurotypical comparison groups. Our three hypotheses were that firstly, self-appraisal measures of depression and anxiety will differentiate the neurotypical group from the cohorts with social impairment. Second, the neurodevelopmental cohort will be distinguished from the SAD group on measures of attention, information processing, social cognition, EF, and visuomotor performance and third, the ASD and EP groups will be differentiated based on their performance on tasks of complex attention.
Our results showed that a reduced set of assessment measures differentiated between the comparison groups with good discriminative ability (AUC ≥ 0.7 and Brier score = 0.14–0.24). Our first hypothesis was confirmed in that depression, anxiety, and stress discriminated the combined social impairment cohort from the comparison control group. Two measures drawn from the social cognition and learning/memory domains (social skill and visuospatial short-term memory) complemented this profile. Our second hypothesis received partial support. Three of the predicted five cognitive domains (visual learning, social cognition, and EF), featured in the optimized profile discriminating between the neurodevelopmental groups and the SAD group. Depression was the other distinguishing feature. Finally, contrary to our third hypothesis, psychomotor speed rather than complex attention distinguished between the EP and ASD groups. Taken together, our research outcomes support and extend literature findings on distinguishing features of the SAD and neurodevelopmental (ASD/EP) groups. The results are particularly compelling given the high discriminative performance of the optimized profiles that emerged from an extensive battery across multiple cognitive domains and affective states.
The first finding of interest is that other cognitive domains in addition to EF and social cognition featured in the optimized profiles. The learning/memory domain measure of visuospatial memory contributed to the combined social impairment cohort versus control discriminating profile. This is surprising given that cognitive function in SAD (57) is generally intact, and thus not expected to differentiate this cohort from a neurotypical control group. The finding suggests that our combined social impairment cohort shares atypicalities in maintaining visual information in short term memory. There is evidence of reduced visual working memory capacity in EP (109) and ASD (110, 111) and our findings may in part reflect this. The shared profile with SAD however, points to more complex processes. A number of cognitive models predict that anxiety attenuates cognitive control and impairs working memory processes (112) including visual working memory (113). Our combined social impairment cohort is characterized by high levels of anxiety, and our findings may reflect the influence of anxiety on executive control.
Measures from learning, attention, and psychomotor speed domains featured in the optimized profiles that discriminated between clinical cohorts. Visual associative learning contributed to discriminating the neurodevelopmental from the SAD cohort. A closer examination of this profile indicated that although all groups were comparable on overall visual learning performance, the neurodevelopmental cohort made more errors. This may reflect impaired processes specific to EP and ASD including impaired visual working memory (109, 110) and slow processing speed (114, 115). Attentional processes were the most salient features that discriminated the EP group from the combined ASD/SAD cohort and EP from the SAD groups. Attentional neural circuitry in EP is clearly impaired in the course of illness (30) and indicates that it may have a unique role in early detection and differentiation. Psychomotor processing speed was the only distinguishing feature discriminating between EP and ASD groups. Research supports that processing speed is impaired in both groups (34, 114) however, different patterns of reaction time changes may apply. There is some evidence that processing speed in EP/SCH deteriorates in later age (116) whilst in ASD, processing speed has matured by adolescence (117) and is significantly impaired compared to neurotypical controls (114). The discriminating profile identified here may reflect different trajectory changes. The absence of measures from other cognitive domains in the EP/ASD comparison support that these two groups have a shared phenotype across most cognitive domains.
The second finding of interest is that phonemic fluency was the only EF measure that contributed to a profile discriminating between SAD and the neurodevelopmental cohort. Phonemic fluency performance is thought to be positively associated with intact frontal lobe function (118) and results may indicate frontal lobe alterations in EP (67) and ASD (119). Given that impairment in EF is noted for both EP (22) and ASD (36, 120) cohorts, greater prominence of EF measures would be expected. The limited role of our other EF measures in differentiating between the clinical groups suggests EF may have greater relevance as a transdiagnostic dimension of neurodevelopment (121).
Social cognition was a distinguishing feature for a number of optimized profiles. These measures featured in all profiles that included participants diagnosed with ASD, except for the ASD/EP direct comparison. Self-appraisals for social skill (a sub-scale of the EQ questionnaire that measures difficulty in social situations), differentiated the clinical cohort from the control group. Co-morbidity with SAD has been reported for each of the EP (122) and ASD (59) groups, and our finding of a shared profile feature likely reflects this. The neurodevelopmental cohort was distinguished from the SAD group on measures of basic emotion recognition (RMET task), identifying emotions in the absence of salient cues (movie stills task) and, in experiencing an appropriate emotion in response to another (self-appraisal of emotional reactivity/empathy). Finally, the ASD versus SAD profile distinguished between the two groups on the overall level of empathy (EQ questionnaire). These findings highlight the salience of social cognition in the neurodevelopmental cohort and particularly for the ASD group. Considered together with the limited prominence of EF features despite known EF deficits, it suggests that social cognition is a more important domain for discriminating the ASD group from other cohorts.
The prominence of mental health features (depression, anxiety, and stress) in the profile discriminating between the combined social impairment cohort and the control group, reflects the high levels of co-morbid depression (61, 64) and anxiety (122) reported for ASD, EP, and SAD. The inclusion of depression and stress self-appraisals in discriminating between the three clinical cohorts warrants further discussion and suggests that nuanced differences differentiate between the groups. The SAD group reported the highest levels of depression in our clinical cohort and EP the lowest levels of stress. Depression was the only affective state that discriminated the SAD versus neurodevelopmental cohort. This may reflect the high levels of co-morbid depression characterizing SAD (61). The lower levels of stress differentiating the EP group from ASD/SAD may reflect differences in symptom severity levels on presentation to our services. Acute positive psychotic symptoms in the EP cohort were controlled prior to inclusion in our services. The ASD and SAD participants however, would be experiencing a more acute profile of their respective symptoms. This may translate to the lower levels of distress reported by the EP group. Alternatively, lower stress in EP may reflect different levels of insight. There is research support that individuals with EP, (particularly those with more impaired cognitive function) have lower levels of insight (123) and may therefore report lower levels of stress.
Limitations
There are a number of limitations in our study. First, although the sample size used in this study was larger than the suggested sample sizes of 75 to 100 for reasonable precision (4) the relatively small sample size of our cohort may reduce the parameters for trainability and cross-validations of our data. A larger sample size would be of benefit to further research. We also acknowledge the resources required to collect the detailed data we have. This is one of the largest studies with detailed information in the field to date. Second, our findings can only be attributed to individuals without intellectual disability, as we did not include any participants with an IQ below 70. Third, our findings include a number of features based on self-appraisals, and there is some question whether self-report appraisals by individuals with ASD are comparable to other cohorts (19, 20). Fourth, a number of participants in the diagnostic groups were being treated with medication, however, we were not able to control for medication use in this study. Fifth, we used 10-fold cross-validation to evaluate the classification performance of models and to identify the discriminating profiles between clinical, neurodevelopmental, and neurotypical comparison groups. Although this approach is considered as the most robust resampling technique to assess the accuracy and generalizability of models (124), the need for a more rigorous approach (external validation) has been emphasized to ensure the model generalizability (125). The present findings, therefore, need to be replicated in future studies with an independent large test set of completely unseen data in order to assess the generalizability of our ML models.
Conclusions
The optimized profiles identified in our study highlight the importance of evaluating multiple cognitive domains when determining discriminating profiles between clinical groups. Further, they demonstrate that our combined social impairment cohort (ASD, EP, and SAD) is characterized by both shared and discriminating features. This has implications for diagnostic, intervention, and remediation strategies. The discriminating profiles can thus facilitate differential diagnosis particularly when clinical cohorts are characterized by comorbid mental health conditions and shared phenotypes. Conversely, the shared profile features, provide a framework for identifying transdiagnostic dimensions for intervention and remediation programs. The unique discriminating features (attention and empathy) that respectively characterized our EP and ASD cohorts potentially identify key target areas for early intervention programs. To-date there has been promising research on the effectiveness of intervention programs in improving social and non-social cognition in populations with ASD (126), EP (127) and SCH (128). In a study investigating cognitive support training in early psychosis (127) improvements were identified, however it was uncertain whether these were restorative or compensatory in nature. In cohorts with ASD, a recent study identified that higher levels of cognitive empathy mediated the positive influence of affective empathy on personal well-being (129). The researchers suggested that training programs on cognitive empathy could contribute to improvements in quality of life in ASD. Taken together these findings suggest that early intervention programs that target attention and empathy in the respective cohorts could contribute to improved functioning and potentially attenuation of symptoms.
To our knowledge, this is the first study that utilized measures across multiple cognitive domains and affective states. Our findings provide a framework for further research on shared and differentiating profiles of neurodevelopmental cohorts and cohorts characterized by social impairment.
Data Availability Statement
The datasets generated for this study are available on request to the corresponding author.
Ethics Statement
The studies involving human participants were reviewed and approved by The University of Sydney Ethics Committee. The patients/participants provided their written informed consent to participate in this study.
Author Contributions
ED—conceptual design, literature review, data collection, manuscript preparation, manuscript revision. SP—conceptual design, data analysis, manuscript preparation, manuscript revision. NH—conceptual design, data analysis, manuscript preparation, manuscript revision. KP—data collection, manuscript revision. YS—manuscript revision. SN—manuscript revision. ET—data collection. IH—manuscript revision. AG—conceptual design, manuscript revision.
Funding
This work was supported by the National Health and Medical Research Council (NHMRC) postgraduate scholarship to ED [GNT1056587], an NHMRC Australian Fellowship [APP 511921] to IH, and ARC Linkage Project grants [LP110100513; LP110200562], a National Health and Medical Research Council Career Development Fellowship [APP1061922], and a Project Grant [1043664; 1125449] to AG.
Conflict of Interest
IH is a Commissioner in Australia's new National Mental Health Commission from 2012. He was a director of headspace: the national youth mental health foundation until January 2012. He was previously the chief executive officer (till 2003) and clinical adviser (till 2006) of beyondblue, an Australian National Depression Initiative. He is the Co-Director, Health and Policy at the Brain and Mind Centre that operates two early-intervention youth services under contract to headspace. He has led a range of community-based and pharmaceutical industry-supported depression awareness and education and training programs. He has led projects for health professionals and the community supported by governmental, community agency and pharmaceutical industry partners (Wyeth, Eli Lily, Servier, Pfizer, AstraZeneca) for the identification and management of depression and anxiety. He has received honoraria for presentations of his own work at educational seminars supported by a number of non-government organisations and the pharmaceutical industry (including Servier, Pfizer, AstraZeneca and Eli Lilly). He is a member of the Medical Advisory Panel for Medibank Private and also a Board Member of Psychosis Australia Trust. He leads an investigator-initiated study of the effects of agomelatine on circadian parameters (supported in part by Servier) and has participated in a multicentre clinical trial of the effects of agomelatine on sleep architecture in depression and a Servier-supported study of major depression and sleep disturbance in primary care settings.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The manuscript formed part of a chapter for the Doctor of Philosophy dissertation of the first author, submitted on 19 December 2018, University of Sydney (130).
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2020.00545/full#supplementary-material
References
1. Chekroud AM. Bigger data, harder questions—opportunities throughout mental health care. JAMA Psychiatry (2017) 74(12):1183–4. doi: 10.1001/jamapsychiatry.2017.3333
2. Huys QM. Advancing clinical improvements for patients using the theory-driven and data-driven branches of computational psychiatry. JAMA Psychiatry (2018) 75(3):225–6. doi: 10.1001/jamapsychiatry.2017.4246
3. Kanchanatawan B, Sriswasdi S, Thika S, Stoyanov D, Sirivichayakul S, Carvalho AF, et al. Towards a new classification of stable phase schizophrenia into major and simple neuro-cognitive psychosis: Results of unsupervised machine learning analysis. J Eval Clin Pract (2018) 24(4):879–91. doi: 10.1111/jep.12945
4. Beleites C, Neugebauer U, Bocklitz T, Krafft C, Popp J. Sample size planning for classification models. Anal chim Acta (2013) 760:25–33. doi: 10.1016/j.aca.2012.11.007
5. Monuteaux MC, Stamoulis C. Machine Learning: A Primer for Child Psychiatrists. J Am Acad Child Adolesc Psychiatry (2016) 55(10):835. doi: 10.1016/j.jaac.2016.07.766
6. Bone D, Bishop Somer L, Black Matthew P, Goodwin Matthew S, Lord C, Narayanan Shrikanth S. Use of machine learning to improve autism screening and diagnostic instruments: effectiveness, efficiency, and multi-instrument fusion. J Child Psychol Psychiatry (2016) 57(8):927–37. doi: 10.1111/jcpp.12559
7. Duda M, Haber N, Daniels J, Wall DP. Crowdsourced validation of a machine-learning classification system for autism and ADHD. Trans Psychiatry (2017) 7:e1133. doi: 10.1038/tp.2017.86
8. Grisanzio KA, Goldstein-Piekarski AN, Wang M, Rashed Ahmed AP, Samara Z, Williams LM. Transdiagnostic symptom clusters and associations with brain, behavior, and daily function in mood, anxiety, and trauma disorders. JAMA Psychiatry (2018) 75(2):201–9. doi: 10.1001/jamapsychiatry.2017.3951
9. Bishop-Fitzpatrick L, Movaghar A, Greenberg JS, Page D, DaWalt LS, Brilliant MH, et al. Using machine learning to identify patterns of lifetime health problems in decedents with autism spectrum disorder. Autism Res (2018) 11(8):1120–8. doi: 10.1002/aur.1960
10. Anney RJL, Ripke S, Anttila V, Grove J, Holmans P, Huang H, et al. Meta-analysis of GWAS of over 16,000 individuals with autism spectrum disorder highlights a novel locus at 10q24.32 and a significant overlap with schizophrenia. Mol Autism (2017) 8(1):21. doi: 10.1186/s13229-017-0137-9
11. Tin LNW, Lui SSY, Ho KKY, Hung KSY, Wang Y, Yeung HKH, et al. High-functioning autism patients share similar but more severe impairments in verbal theory of mind than schizophrenia patients. psychol Med (2017) 48(8):1264–73. doi: 10.1017/S0033291717002690
12. Addington J, Brooks BL, Addington D. Cognitive functioning in first episode psychosis: initial presentation. Schizophr Res (2003) 62(1):59–64. doi: 10.1016/S0920-9964(02)00340-7
13. Goldstein G, Minshew NJ, Allen DN, Seaton BE. High-functioning autism and schizophrenia A comparison of an early and late onset neurodevelopmental disorder. Arch Clin Neuropsychol (2002) 17(5):461–75. doi: 10.1093/arclin/17.5.461
14. Chisholm K, Lin A, Abu-Akel A, Wood SJ. The association betweeen autism and schizophrenia spectrum disorders: a review of eight alternate models of co-occurrence. Neurosci Biobehavioural Rev (2015) 55:173–83. doi: 10.1016/j.neubiorev.2015.04.012
15. Park MTM, Raznahan A, Shaw P, Gogtay N, Lerch JP, Chakravarty MM. Neuroanatomical phenotypes in mental illness: identifying convergent and divergent cortical phenotypes across autism, ADHD and schizophrenia. J Psychiatry Neurosci : JPN (2018) 43(3):201–12. doi: 10.1503/jpn.170094
16. DiLalla LF, McCrary M, Diaz E. A review of endophenotypes in schizophrenia and autism: The next phase for understanding genetic etiologies. Am J Med Genet Part C: Semin Med Genet (2017) 175(3):354–61. doi: 10.1002/ajmg.c.31566
17. O'Connell KS, McGregor NW, Lochner C, Emsley R, Warnich L. The genetic architecture of schizophrenia, bipolar disorder, obsessive-compulsive disorder and autism spectrum disorder. Mol Cell Neurosci (2018) 88:300–7. doi: 10.1016/j.mcn.2018.02.010
18. Lugo Marín J, Alviani Rodríguez-Franco M, Mahtani Chugani V, Magán Maganto M, Díez Villoria E, Canal Bedia R. Prevalence of schizophrenia spectrum disorders in average-iq adults with autism spectrum disorders: a meta-analysis. J Autism Dev Disord (2018) 48(1):239–50. doi: 10.1007/s10803-017-3328-5
19. Pepper KL, Demetriou EA, Park SH, Song YC, Hickie IB, Cacciotti-Saija C, et al. Autism, early psychosis, and social anxiety disorder: understanding the role of social cognition and its relationship to disability in young adults with disorders characterized by social impairments. Trans Psychiatry (2018) 8(1):233–. doi: 10.1038/s41398-018-0282-8
20. Demetriou EA, Song CY, Park SH, Pepper KL, Naismith SL, Hermens DF, et al. Autism, Early Psychosis, and Social Anxiety Disorder: a transdiagnostic examination of executive function cognitive circuitry and contribution to disability. Transl Psychiatry (2018) 8(1):200. doi: 10.1038/s41398-018-0193-8
21. Happé FG, Mansour H, Barrett P, Brown T, Abbott P, Charlton RA. Demographic and Cognitive Profile of Individuals Seeking a Diagnosis of Autism Spectrum Disorder in Adulthood. J Autism Dev Disord (2016) 46(11):3469–80. doi: 10.1007/s10803-016-2886-2
22. Seidman LJ, Shapiro DI, Stone WS, Woodberry KA, Ronzio A, Cornblatt BA. Association of neurocognition with transition to psychosis: Baseline functioning in the second phase of the north american prodrome longitudinal study. JAMA Psychiatry (2016) 73(12):1239–48. doi: 10.1001/jamapsychiatry.2016.2479
23. Fusar-Poli P, Bonoldi I, Yung AR, Borwadt S, Kempton MJ, Valmaggia L, et al. Predicting psychosis: Meta-analysis of transition outcomes in individuals at high clinical risk. Arch Gen Psychiatry (2012) 69(3):220–9. doi: 10.1001/archgenpsychiatry.2011.1472
24. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Washington, DC: American Psychiatric Association; (2013).
25. Baggetta P, Alexander P A. Conceptualization and operationalization of executive function. Mind Brain Educ (2016) 10(1):10–33. doi: 10.1111/mbe.12100
26. Demetriou EA, DeMayo MM, Guastella AJ. Executive Function in Autism Spectrum Disorder: History, Theoretical Models, Empirical Findings, and Potential as an Endophenotype. Front Psychiatry (2019) 10(753):1–17. doi: 10.3389/fpsyt.2019.00753
27. Lezak MD, Howieson DB, Bigler ED, Tranel D. Neuropsychological Assessment. (5th ed), Oxford: Oxford University Press. (2012).
28. Chien YL, Gau SSF, Shang CY, Chiu YN, Tsai WC, Wu YY. Visual memory and sustained attention impairment in youths with autism spectrum disorders. psychol Med (2015) 45(11):2263–73. doi: 10.1017/S0033291714003201
29. Pina-Camacho L, Garcia-Prieto J, Parellada M, Castro-Fornieles J, Gonzalez-Pinto AM, Bombin I, et al. Predictors of schizophrenia spectrum disorders in early-onset first episodes of psychosis: a support vector machine model. Eur Child Adolesc Psychiatry (2015) 24(4):427–40. doi: 10.1007/s00787-014-0593-0
30. Silton RL, Heller W, Towers DN, Engels AS, Spielberg JM, Edgar JC, et al. The time course of activity in dorsolateral prefrontal cortex and anterior cingulate cortex during top-down attentional control. NeuroImage (2010) 50(3):1292–302. doi: 10.1016/j.neuroimage.2009.12.061
31. Bowler DM, Limoges E, Mottron L. Different verbal learning strategies in autism spectrum disorder: Evidence from the Rey Auditory Verbal Learning Test. J Autism Dev Disord (2009) 39(6):910–5. doi: 10.1007/s10803-009-0697-4
32. Lever AG, Geurts HM. Age-related differences in cognition across the adult lifespan in autism spectrum disorder. Autism Res (2016) 9(6):666–76. doi: 10.1002/aur.1545
33. Carrión RE, Walder DJ, Auther AM, McLaughlin D, Zyla HO, Adelsheim S, et al. From the psychosis prodrome to the first-episode of psychosis: No evidence of a cognitive decline. J Psychiatr Res (2018) 96:231–8. doi: 10.1016/j.jpsychires.2017.10.014
34. Ayesa-Arriola R, Setién-Suero E, Neergaard KD, Ferro A, Fatjó-Vilas M, Ríos-Lago M, et al. Evidence for trait related Theory of Mind impairment in First Episode Psychosis patients and its relationship with processing speed: A 3 Year follow-up study. Front Psychol (2016) 7(592):1–8. doi: 10.3389/fpsyg.2016.00592
35. Williams DL, Minshew NJ, Goldstein G. Further understanding of complex information processing in verbal adolescents and adults with autism spectrum disorders. Autism (2015) 19(7):859–67. doi: 10.1177/1362361315586171
36. Lai CLE, Lau Z, Lui SSY, Lok E, Tam V, Chan Q, et al. Meta-analysis of neuropsychological measures of executive functioning in children and adolescents with high-functioning autism spectrum disorder. Autism Res (2017) 10(5):911–39. doi: 10.1002/aur.1723
37. Baron-Cohen S, Leslie AM, Frith U. Does the autistic child have a “theory of mind” ? Cogn. (1985) 21(1):37–46. doi: 10.1016/0010-0277(85)90022-8
38. Uljarevic M, Hamilton A. Recognition of emotions in autism: A formal meta-analysis. J Autism Dev Disord (2013) 43(7):1517–26. doi: 10.1007/s10803-012-1695-5
39. Healey KM, Bartholomeusz CF, Penn DL. Deficits in social cognition in first episode psychosis: A review of the literature. Clin Psychol Rev (2016) 50:108–37. doi: 10.1016/j.cpr.2016.10.001
40. Murray K, Johnston K, Cunnane H, Kerr C, Spain D, Gillan N, et al. A new test of advanced theory of mind: The “Strange Stories Film Task” captures social processing differences in adults with autism spectrum disorders. Autism Res (2017) 10(6):1120–32. doi: 10.1002/aur.1744
41. Ozonoff S, McEvoy RE. A longitudinal study of executive function and theory of mind development in autism. Dev Psychopathol (1994) 6:415–31. doi: 10.1017/S0954579400006027
42. Burton CZ, Harvey PD, Patterson TL, Twamley EW. Neurocognitive insight and objective cognitive functioning in schizophrenia. Schizophr Res (2016) 171(1):131–6. doi: 10.1016/j.schres.2016.01.021
43. Geurts HM, Bergh SFWM, Ruzzano L. Prepotent Response Inhibition and Interference Control in Autism Spectrum Disorders: Two Meta-Analyses. Autism Res (2014) 7(4):407–20. doi: 10.1002/aur.1369
44. Demetriou EA, Lampit A, Quintana DS, Naismith SL, Song YJC, Pye JE, et al. Autism spectrum disorders: a meta-analysis of executive function. Mol Psychiatry (2018) 23(5):1198–204. doi: 10.1038/mp.2017.75
45. Kanchanatawan B, Thika S, Anderson G, Galecki P, Maes M. Affective symptoms in schizophrenia are strongly associated with neurocognitive deficits indicating disorders in executive functions, visual memory, attention and social cognition. Prog Neuropsychopharmacol Biol Psychiatry (2018) 80:168–76. doi: 10.1016/j.pnpbp.2017.06.031
46. Deshpande G, Libero L, Sreenivasan K, Deshpande H, Kana R. Identification of neural connectivity signatures of autism using machine learning. Front Hum Neurosci (2013) 7(670):1–15. doi: 10.3389/fnhum.2013.00670
47. Eack SM, Bahorik AL, McKnight SAF, Hogarty SS, Greenwald DP, Newhill CE, et al. Commonalities in social and non-social cognitive impairments in adults with autism spectrum disorder and schizophrenia. Schizophr Res (2013) 148(1):24–8. doi: 10.1016/j.schres.2013.05.013
48. Pinkham AE, Morrison KE, Penn DL, Harvey PD, Kelsven S, Ludwig K, et al. Comprehensive comparison of social cognitive performance in autism spectrum disorder and schizophrenia. Psychol Med (2019), 1–9. doi: 10.1017/S0033291719002708
49. Sasson NJ, Pinkham AE, Weittenhiller LP, Faso DJ, Simpson C. Context Effects on Facial Affect Recognition in Schizophrenia and Autism: Behavioral and Eye-Tracking Evidence. Schizophr Bull (2015) 42(3):675–83. doi: 10.1093/schbul/sbv176
50. Bliksted V, Ubukata S, Koelkebeck K. Discriminating autism spectrum disorders from schizophrenia by investigation of mental state attribution on an on-line mentalizing task: A review and meta-analysis. Schizophr Res (2016) 171(1):16–26. doi: 10.1016/j.schres.2016.01.037
51. Eack SM, Wojtalik JA, Keshavan MS, Minshew NJ. Social-cognitive brain function and connectivity during visual perspective-taking in autism and schizophrenia. Schizophr Res (2017) 183:102–9. doi: 10.1016/j.schres.2017.03.009
52. Stanfield AC, Philip RCM, Whalley H, Romaniuk L, Hall J, Johnstone EC, et al. Dissociation of Brain Activation in Autism and Schizotypal Personality Disorder During Social Judgments. Schizophr Bull (2017) 43(6):1220–8. doi: 10.1093/schbul/sbx083
53. Ciaramidaro A, Bölte S, Schlitt S, Hainz D, Poustka F, Weber B, et al. Schizophrenia and Autism as Contrasting Minds: Neural Evidence for the Hypo-Hyper-Intentionality Hypothesis. Schizophr Bull (2014) 41(1):171–9. doi: 10.1093/schbul/sbu124
54. Ciaramidaro A, Bölte S, Schlitt S, Hainz D, Poustka F, Weber B, et al. Transdiagnostic deviant facial recognition for implicit negative emotion in autism and schizophrenia. Eur Neuropsychopharmacol (2018) 28(2):264–75. doi: 10.1016/j.euroneuro.2017.12.005
55. Crespi B, Badcock C. Psychosis and autism as diametrical disorders of the social brain. Behav Brain Sci (2008) 31(3):241–61. doi: 10.1017/S0140525X08004214
56. Volkmar FR, Lord C, Bailey A, Schultz RT, Klin A. Autism and pervasive developmental disorders. J Child Psychol Psychiatry (2004) 45(1):135–70. doi: 10.1046/j.0021-9630.2003.00317.x
57. O'Toole MS, Pedersen AD. A systematic review of neuropsychological performance in social anxiety disorder. Nord J Psychiatry (2011) 65(3):147–61. doi: 10.3109/08039488.2011.565801
58. Michail M, Birchwood M. Social anxiety disorder in first-episode psychosis: incidence, phenomenology and relationship with paranoia. Br J Psychiatry (2018) 195(3):234–41. doi: 10.1192/bjp.bp.108.053124
59. Maddox BB, White SW. Comorbid social anxiety disorder in adults with autism spectrum disorder. J Autism Dev Disord (2015) 45(12):3949–60. doi: 10.1007/s10803-015-2531-5
60. Birchwood M. Pathways to emotional dysfunction in first-episode psychosis. Br J Psychiatry (2018) 182(5):373–5. doi: 10.1192/bjp.182.5.373
61. Adams GC, Balbuena L, Meng X, Asmundson GJG. When social anxiety and depression go together: A population study of comorbidity and associated consequences. J Affect Disord (2016) 206:48–54. doi: 10.1016/j.jad.2016.07.031
62. Stein MB, Tancer ME, Gelernter CS, Vittone BJ, Uhde TW. Major depression in patients with social phobia. Am J Psychiatry (1990) 147(5):637–9. doi: 10.1176/ajp.147.5.637
63. Van Ameringen M, Mancini C, Styan G, Donison D. Relationship of social phobia with other psychiatric illness. J Affect Disord (1991) 21(2):93–9. doi: 10.1016/0165-0327(91)90055-W
64. Dai J, Du X, Yin G, Zhang Y, Xia H, Li X, et al. Prevalence, demographic and clinical features of comorbid depressive symptoms in drug naïve patients with schizophrenia presenting with first episode psychosis. Schizophr Res (2018) 193:182–7. doi: 10.1016/j.schres.2017.06.029
65. Hofvander B, Delorme R, Chaste P, Nydén A, Wentz E, Ståhlberg O, et al. Psychiatric and psychosocial problems in adults with normal-intelligence autism spectrum disorders. BMC Psychiatry (2009) 9(1):35. doi: 10.1186/1471-244X-9-35
66. Lugnegard T, Hallerback MU, Gillberg C. Psychiatric comorbidity in young adults with a clinical diagnosis of Asperger syndrome. Res Dev Disabil (2011) 32(5):1910–7. doi: 10.1016/j.ridd.2011.03.025
67. Wood SJ, Pantelis C, Velakoulis D, Yücel M, Fornito A, McGorry PD. Progressive changes in the development toward schizophrenia: studies in subjects at increased symptomatic risk. Schizophr Bull (2008) 34(2):322–9. doi: 10.1093/schbul/sbm149
68. Lord C, Rutter M, DiLavore P, Risi S, Gotham K. Autism diagnostic observation schedule-2nd edition (ADOS-2). Los Angeles, CA: Western Publication Services (2012).
69. Brown TA, DiNardo PA, Barlow DH. Anxiety Disorders Interview Schedule for DSM-IV. 198 Madison Avenue, New York, New York, 10016: Oxford University Press; (1994).
70. First MB, Gibbon M. The Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) and the Structured Clinical Interview for DSM-IV Axis II Disorders (SCID-II). In: Comprehensive handbook of psychological assessment, Vol 2: Personality assessment. Hoboken, NJ, US: John Wiley & Sons Inc; (2004). p. 134–43.
71. Andresen NC, Flaum M, Arndt S, Alliger R, Swayze VW. Positive and negative symptoms: Assessment and Validity. In: Marneros A., Andreasen N.C., Tsuang M.T. (eds) Negative Versus Positive Schizophrenia. Berlin, Heidelberg: Springer. (1991). doi: 10.1007/978-3-642-76841-5_3
72. Wechsler D. Wechsler Abbreviated Scale of Intelligence (WASI) Manual. San Antonio, Texas: Psychological Corportation; (1999).
73. Wechsler D. Wechsler Test of Adult Reading. San Antonio, Texas: Psychological Corporation; (2001).
74. Lovibond PF, Lovibond SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther (1995) 33(3):335–43. doi: 10.1016/0005-7967(94)00075-U
75. Heimberg RG, Mueller GP, Holt CS, Hope DA, Liebowitz MR. Assessment of anxiety in social interaction and being observed by others: The Social Interaction Anxiety Scale and the Social Phobia Scale. Behav Ther (1992) 23(1):53–73. doi: 10.1016/S0005-7894(05)80308-9
76. Allison C, Auyeung B, Baron-Cohen S. Toward brief “Red Flags” for autism screening: The Short Autism Spectrum Quotient and the Short Quantitative Checklist for Autism in toddlers in 1,000 cases and 3,000 controls [corrected]. J Am Acad Child Adolesc Psychiatry (2012) 51(2):202–12 e7. doi: 10.1016/j.jaac.2011.11.003
77. Constantino JN. Social Responsiveness Scale. New York, NY: Springer. 2nd ed. (2012). doi: 10.1007/978-1-4419-1698-3
78. Baron-Cohen S, Wheelwright S. The Empathy Quotient: An Investigation of Adults with Asperger Syndrome or High Functioning Autism, and Normal Sex Differences. J Autism Dev Disord (2004) 34(2):163–75. doi: 10.1023/B:JADD.0000022607.19833.00
79. Stone VE, Baron-Cohen S, Knight RT. Frontal Lobe Contributions to Theory of Mind. J Cogn Neurosci (1998) 10(5):640–56. doi: 10.1162/089892998562942
80. Young A, Perrett D, Calder A, Sprengelmeyer R, Ekman P. Facial Expressions of Emotion: Stimuli and Tests (FEEST). (Thames Valley Company, Edmunds, UK, 2002). Edmunds, UK: Thames Valley Company; (2002).
81. Adolphs R, Tranel D. Amygdala damage impairs emotion recognition from scenes only when they contain facial expressions. Neuropsychologia (2003) 41(10):1281–9. doi: 10.1016/S0028-3932(03)00064-2
82. Langdon R, Michie PT, Ward PB, McConaghy N, Catts SV, Coltheart M. Defective Self and/or Other Mentalising in Schizophrenia: A Cognitive Neuropsychological Approach. Cogn Neuropsychiatry (1997) 2(3):167–93. doi: 10.1080/135468097396324
83. Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I. The “Reading the Mind in the Eyes” Test Revised Version: A Study with Normal Adults, and Adults with Asperger Syndrome or High-functioning Autism. J Child Psychol Psychiatry Allied Disciplines (2001) 42(2):241–51. doi: 10.1111/1469-7610.00715
84. Roth RM, Isquith PK, Gioia G. BRIEF-A: Behavior Rating Inventory of Executive Function-Adult Version. Lutz, FL 33549: PAR; (2005).
85. Ruff RM, Light RH, Parker SB, Levin HS. Benton controlled oral word association test: Reliability and updated norms. Arch Clin Neuropsychol (1996) 11(4):329–38. doi: 10.1093/arclin/11.4.329
86. Tombaugh TN. Trail Making Test A and B: Normative data stratified by age and education. Arch Clin Neuropsychol (2004) 19(2):203–14. doi: 10.1016/S0887-6177(03)00039-8
87. De Luca CR, Wood SJ, Anderson V, Buchanan J-A, Proffitt TM, Mahony K, et al. Normative data from the CANTAB. I: Development of executive function over the lifespan. J Clin Exp Neuropsychol (2003) 25(2):242–54. doi: 10.1076/jcen.25.2.242.13639
88. Wechsler D. Wechsler Memory Scale. 3rd ed. San Antonio, Texas: The Psychological Corporation; (1997).
89. McMinn MR, Wiens AN, Crossen JR. Rey auditory-verbal learning test: Development of norms for healthy young adults. Clin Neuropsychologist. (1988) 2(1):67–87. doi: 10.1080/13854048808520087
90. van Buuren S, Groothuis-Oudshoorn K. mice: Multivariate Imputation by Chained Equations in R. J Stat Software (2011) 45(3):1–67. doi: 10.18637/jss.v045.i03
91. Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In volume 14. Int Joint Conf Artif Intell (1995) 14:1137–45. doi: 10.5555/1643031.1643047
92. Moss H, Leslie D, Rayson P. (2018). Using J-K-fold Cross Validation To Reduce Variance When Tuning NLP Models. Santa Fe, New Mexico USA: Association for Computational Linguistics, pp. 2978–89.
93. Wolpert DH, Macready WG. No free lunch theorems for optimization. IEEE Trans evol comput (1997) 1(1):67–82. doi: 10.1109/4235.585893
94. Calle ML, Urrea V, Boulesteix AL, Malats N. AUC-RF: A New Strategy for Genomic Profiling with Random Forest. Hum Heredity (2011) 72(2):121–32. doi: 10.1159/000330778
95. Kursa MB, Rudnicki WR. Feature selection with the Boruta package. J Stat Software (2010) 36(11):1–13. doi: 10.18637/jss.v036.i11
96. Tibshirani R. Regression shrinkage and selection via the Lasso. J R Stat Soc Ser B (Methodological) (1996) 58(1):267–88. doi: 10.1111/j.2517-6161.1996.tb02080.x
97. Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc: Ser B (Stat Method) (2005) 67(2):301–20. doi: 10.1111/j.1467-9868.2005.00503.x
98. Chipman HA, George EI, McCulloch RE. BART: Bayesian additive regression trees. Ann Appl Stat (2010) 4(1):266–98. doi: 10.1214/09-AOAS285
99. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. . Radiol (1982) 143(1):29–36. doi: 10.1148/radiology.143.1.7063747
100. Cortes C, Mohri M. AUC optimization vs. error rate minimization. Proceedings of the 2003 Conference, NIPS 2003 (Advances in Neural Information Processing Systems) Neural Information Processing Systems Foundation (2004), 313–20.
101. Fawcett T. ROC graphs: Notes and practical considerations for researchers. Mach Learn (2004) 31(1):1–38.
102. Ling CX, Huang J, Zhang H. AUC: a better measure than accuracy in comparing learning algorithms. In: . Conference of the canadian society for computational studies of intelligence. Springer, Berlin, Heidelberg. Berlin, Heidelberg: Springer; (2003). p. 329–41.
103. Ling CX, Huang J, Zhang H. AUC: a statistically consistent and more discriminating measure than accuracy. JCAI. (2003) 3:519–24. doi: 10.5555/1630659.1630736
104. Hosmer J, Lemeshow S, Sturdivant R. Applied Logistic Regression. New Jersey: John Wiley and Sons. (2013).
105. Rufibach K. Use of Brier score to assess binary predictions. J Clin Epidemiol (2010) 63(8):938–9. doi: 10.1016/j.jclinepi.2009.11.009
106. Kursa MB, Rudnicki WR. Feature Selection with the Boruta Package. 2010. (2010) 36(11):13. doi: 10.18637/jss.v036.i11
108. Hastie T, Tibshirani R. Bayesian Backfitting. Stat Sci (2000) 15(3):196–213. doi: 10.1214/ss/1009212815
109. Wannan CMJ, Bartholomeusz CF, Cropley VL, Van Rheenen TE, Panayiotou A, Brewer WJ, et al. Deterioration of visuospatial associative memory following a first psychotic episode: a long-term follow-up study. psychol Med (2017) 48(1):132–41. doi: 10.1017/S003329171700157X
110. Cardillo R, Menazza C ICM. Visuoconstructive abilities and visuospatial memory in autism spectrum disorder without intellectual disability: Is the role of local bias specific to the cognitive domain tested? Neuropsychology (2018) 32(7):822–34. doi: 10.1037/neu0000472
111. Williams DL, Goldstein G, Carpenter PA, Minshew NJ. Verbal and spatial working memory in autism. J Autism Dev Disord (2005) 35(6):747. doi: 10.1007/s10803-005-0021-x
112. Bishop S. Trait anxiety and impoverished prefronatl control of attention. Nat Neurosci (2009) 12(1):92–8. doi: 10.1038/nn.2242
113. Eysenck M, Payne S, Derakshan N. Trait anxiety, visuospatial processing, and working memory. Cogn Emotion (2005) 19(8):1214–28. doi: 10.1080/02699930500260245
114. Haigh SM, Walsh JA, Mazefsky CA, Minshew NJ, Eack SM. Processing speed is impaired in adults with autism spectrum disorder, and relates to social communication abilities. J Autism Dev Disord (2018) 48(8):2653–62. doi: 10.1007/s10803-018-3515-z
115. Liu C-C, Hua M-S, Hwang T-J, Chiu C-Y, Liu C-M, Hsieh MH, et al. Neurocognitive functioning of subjects with putative pre-psychotic states and early psychosis. Schizophr Res (2015) 164(1):40–6. doi: 10.1016/j.schres.2015.03.006
116. Mathias SR, Knowles EEM, Barrett J, Leach O, Buccheri S, Beetham T, et al. The Processing-Speed Impairment in Psychosis Is More Than Just Accelerated Aging. Schizophr Bull (2017) 43(4):814–23. doi: 10.1093/schbul/sbw168
117. Luna B, Doll SK, Hegedus SJ, Minshew NJ, Sweeney JA. Maturation of executive function in autism. Biol Psychiatry (2007) 61(4):474–81. doi: 10.1016/j.biopsych.2006.02.030
118. Tupak SV, Badewien M, Dresler T, Hahn T, Ernst LH, Herrmann MJ, et al. Differential prefrontal and frontotemporal oxygenation patterns during phonemic and semantic verbal fluency. Neuropsychologia (2012) 50(7):1565–9. doi: 10.1016/j.neuropsychologia.2012.03.009
119. Libero LE, DeRamus TP, Lahti AC, Deshpande G, Kana RK. Multimodal neuroimaging based classification of autism spectrum disorder using anatomical, neurochemical, and white matter correlates. Cortex. (2015) 66:46–59. doi: 10.1016/j.cortex.2015.02.008
120. Leung RC, Zakzanis KK. Brief report: cognitive flexibility in autism spectrum disorders: a quantitative review. J Autism Dev Disord (2014) 44(10):2628–45. doi: 10.1007/s10803-014-2136-4
121. McGrath LM, Braaten EB, Doty ND, Willoughby BL, Wilson HK, O'Donnell EH, et al. Extending the ‘cross-disorder' relevance of executive functions to dimensional neuropsychiatric traits in youth. J Child Psychol Psychiatry (2016) 57(4):462–71. doi: 10.1111/jcpp.12463
122. Michail M, Birchwood M. Social anxiety disorder in first-episode psychosis: incidence, phenomenology and relationship with paranoia. Br J Psychiatry (2009) 195(3):234–41. doi: 10.1192/bjp.bp.108.053124
123. Saeedi H, Addington J, Addington D. The association of insight with psychotic symptoms, depression, and cognition in early psychosis: A 3-year follow-up. Schizophr Res (2007) 89(1):123–8. doi: 10.1016/j.schres.2006.09.018
124. Dwyer DB, Falkai P, Koutsouleris N. Machine Learning Approaches for Clinical Psychology and Psychiatry. Annu Rev Clin Psychol (2018) 14:91–118. doi: 10.1146/annurev-clinpsy-032816-045037
125. Taylor JM, Ankerst DP, Andridge RR. Validation of biomarker-based risk prediction models. Clin Cancer Res : an Off J Am Assoc Cancer Res (2008) 14(19):5977–83. doi: 10.1158/1078-0432.CCR-07-4534
126. Holopainen A, de Veld DMJ, Hoddenbach E, Begeer S. Does Theory of Mind Training Enhance Empathy in Autism? J Autism Dev Disord (2019) 49(10):3965–72. doi: 10.1007/s10803-018-3671-1
127. Kidd SA, Herman Y, Virdee G, Bowie CR, Velligan D, Plagiannakos C, et al. A comparison of compensatory and restorative cognitive interventions in early psychosis. Schizophr Res: Cogn (2020) 19:100157. doi: 10.1016/j.scog.2019.100157
128. Turner DT, McGlanaghy E, Cuijpers P, van der Gaag M, Karyotaki E, MacBeth A. A Meta-Analysis of Social Skills Training and Related Interventions for Psychosis. Schizophr Bull (2017) 44(3):475–91. doi: 10.1093/schbul/sbx146
129. Bos J, Stokes MA. Cognitive empathy moderates the relationship between affective empathy and wellbeing in adolescents with autism spectrum disorder. Eur J Dev Psychol (2019) 16(4):433–46. doi: 10.1080/17405629.2018.1444987
Keywords: autism spectrum disorder, cognition, differential diagnosis, early psychosis, machine learning, social anxiety disorder
Citation: Demetriou EA, Park SH, Ho N, Pepper KL, Song YJC, Naismith SL, Thomas EE, Hickie IB and Guastella AJ (2020) Machine Learning for Differential Diagnosis Between Clinical Conditions With Social Difficulty: Autism Spectrum Disorder, Early Psychosis, and Social Anxiety Disorder. Front. Psychiatry 11:545. doi: 10.3389/fpsyt.2020.00545
Received: 26 February 2020; Accepted: 27 May 2020;
Published: 19 June 2020.
Edited by:
Noah James Sasson, The University of Texas at Dallas, United StatesReviewed by:
Joe Bathelt, University of Amsterdam, NetherlandsAhmad Abu-Akel, University of Lausanne, Switzerland
Copyright © 2020 Demetriou, Park, Ho, Pepper, Song, Naismith, Thomas, Hickie and Guastella. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Adam J. Guastella, adam.guastella@sydney.edu.au
†These authors share first authorship