Identifying Clinically and Functionally Distinct Groups Among Healthy Controls and First Episode Psychosis Patients by Clustering on EEG Patterns

Objective: The mismatch negativity (MMN) is considered as a promising biomarker that can inform future therapeutic studies. However, there is a large variability among patients with first episode psychosis (FEP). Also, most studies report a single electrode site and on comparing case–control group differences. Few have taken advantage of the full wealth of multi-channel EEG signals to examine observable patterns. None, to our knowledge, have used machine learning (ML) approaches to investigate neurophysiological derived subgroups with distinct cognitive and functional outcome characteristics. In this study, we applied ML to empirically stratify individuals into homogeneous subgroups based on multi-channel MMN data. We then characterized the functional, cognitive, and clinical profiles of these neurobiologically derived subgroups. We also explored the underlying low frequency range responses (delta, theta, alpha) during MMN.

Methods: Clinical, neurocognitive, functioning data of 33 healthy controls and 20 FEP patients were collected. 90% of the patients had 6-month follow-up data. Neurocognition, social cognition, and functioning measures were assessed using the NCCB Cognitive Battery, the Awareness of Social Inference Test, UCSD Performance-Based Skills Assessment, and Multnomah Community Ability Scale. Symptom severity was collected using the PANSS. MMN amplitude and single-trial derived low frequency activity across 24 frontocentral channels were used as main variables in the ML k-means clustering analyses.

Results: We found a consistent pattern of two distinctive subgroups. We labeled them as “better functioning” and “poorer functioning” clusters, respectively. Each subgroup can be mapped onto either better or poorer clinical, cognitive, and functioning profiles. Also, we identified two subgroups of patients: one showed improved MMN and one showed worsening of MMN over time. Patients with improved MMN had better follow-up clinical, cognitive, and functioning profile than those with worsening MMN. Among the low frequency bands, delta frequency appeared to be the most relevant to the observed MMN responses in all individuals. However, higher delta responses were not necessarily associated with a better functioning profile, suggesting that delta frequency alone may not be useful in clinical characterization.

Conclusions: The ML approach could be a robust tool to explore heterogeneity and facilitate the identification of neurobiological homogeneous subgroups in FEP.

Introduction

Psychosis is one of the most disabling conditions worldwide (1, 2). Early intervention can play a substantial role in improving long-term outcomes, although there is a large variability in treatment responsiveness in the first episode of psychosis (FEP) patients (3). FEP patients demonstrate a wide range of cognitive and neurophysiological impairments and are considerably heterogeneous in the functional outcome trajectories (4–6). Progress is undoubtedly hampered by considerable biological and clinical heterogeneity across FEP: effective treatments are unlikely to advance substantially until disease mechanisms are better understood, and biologically-based objective markers are available to tag the cardinal dysfunction, not the diagnoses or symptoms.

Detecting unexpected stimuli in the environment is a critical function of the auditory system. The mismatch negativity (MMN) is the best-studied electrophysiological marker of deviance detection. It is typically elicited in oddball paradigms and measured as the difference between the event-related potential (ERP) responses to deviant (e.g., duration, pitch) and standard stimuli. The magnitude of MMN ERP is typically greater in the frontocentral electrode sites and peaks between 100 and 250 ms (7, 8). Larger responses (more negative) to rare/deviant stimuli are thought to represent the detection of regularity violation (9–11). Such predominantly automatic (preattentive) process of detecting a “mismatch” between the deviant stimulus and a sensory memory trace might be a critical transitional step from sensory-based processing to the subsequent engagement of higher attentional neural networks necessary for cognitive and psychosocial functioning (12–16).

MMN, particularly the duration-deviant MMN (dMMN), is considered as a promising candidate biomarker that can inform future therapeutic studies of FEP (17–22). Evidence suggests that MMN peak amplitude is a sensitive index of NMDA (15, 23–25) and nicotinic receptor functioning (26). In healthy and chronic schizophrenia (SZ) patients, MMN activity is significantly correlated with distinct domains of cognitive (27–29) and psychosocial, work functioning, and independent living functioning (17, 30–32). Furthermore, MMN deficit increases with the progression of the disease. Meta-analysis found that in FEP patients, the effect size of dMMN deficit is small to medium (Cohen’s d = 0.47) (20); in chronic schizophrenia patients, a systematic increase in effect size was found as a function of illness duration, indicating that dMMN deficit reflects, to some extent, disease progression (33, 34). Although dMMN deficit is reported in FEP patients, the small effect size suggests a large variability among patients. Given that heterogeneity is a key feature of FEP that manifests on clinical, neurobiological, and functioning levels resulting in a substantial barrier to understand disease mechanisms, a data driven approach to stratify FEP patients into homogeneous subgroups would allow a better understanding of the source of variability and biological mechanisms.

Moreover, the relationships between dMMN change over time and its corresponding clinical changes are largely unknown. Longitudinal MMN studies in FEP are rare. Pitch-deviant MMN studies showed that FEP patients with the most impaired MMN amplitudes at baseline showed the most severe disability at follow-up (35) and that MMN was intact at baseline in a majority of FEP patients but worsened at follow-up (36). Identifying subgroups of FEP patients with distinct patterns of dMMN changes over time as well as the corresponding clinical, cognitive, or functioning changes may facilitate early identification of a subgroup of patients at heightened risk for cognitive and functioning decline.

The majority of clinical studies of MMN typically focus on using a single electrode site (Fz) and on comparing case–control group differences. Few have taken advantage of the full wealth of information about brain dynamic processes and observable patterns contained in multi-channel MMN EEG signals. Conventional approaches based on group (case–control) comparisons assume controls and patients as homogeneous populations which does not adequately address the heterogeneity of within-group individual differences. Machine learning (ML) data-driven approaches to address heterogeneity have received renewed interest in partitioning individuals into more homogeneous subgroups (37, 38). The advances in ML approaches make it possible to extract information from complex and high-dimensional data.

In terms of exploring the underlying EEG frequency responses during MMN, previous studies showed that MMN was primarily comprised of lower range frequency evoked oscillations including delta, theta, and alpha (39–42). In controls, theta–alpha frequency was found to be the most significant contributor for MMN, while in patients with SZ spectrum disorders, delta range activities were found to explain the most variance of observed MMN abnormalities. MMN reflects activity primarily in low frequency band, which is thought to depend primarily upon interplay between cortical pyramidal neurons and somatostatin type local circuit GABAergic interneurons (42). However, few studies have examined patients with SZ (41). It is currently unknown to what extent the theta frequency activity plays a role in the generation of MMN in FEP patients. Moreover, it remains unclear to what extent the low frequency in the delta, theta, and alpha range activity is related to cognitive or daily functioning measures; that is, whether abnormal low frequency activity in the delta, theta, and alpha bands is correlated with impaired cognitive or functional measures with respect to the correlations found using MMN amplitude. Finally, the longitudinal change of low frequency range activity in FEP has not been reported in the literature.

In this study, we collected an array of functional, cognitive, clinical, and multi-channel MMN data from a cohort of controls and FEP patients (90% had 6-month follow-up data). We aimed to use an unsupervised ML clustering technique to address several key research questions: i) to stratify individuals into more homogeneous subgroups based on multi-channel MMN activities and examine whether each neurobiologically derived subgroup could be mapped onto a consistent pattern of functional, cognitive, and clinical profile; ii) to stratify FEP patients into distinct clusters based on the change of dMMN over a 6-month period and examine whether patient’s MMN change in each cluster could correspond to a consistent pattern of follow-up functional, cognitive, and clinical profile; iii) to investigate the magnitude of low-frequency (delta, theta, alpha) activities in both controls and patients; iv) to stratify individuals into more homogeneous subgroups based on multi-channel low frequency activity and examine whether delta, theta, and alpha range activities are related to cognitive or daily functioning measures.

We applied data driven strategy, K-means, to first addressed heterogeneity in MMN and low frequency measures among patients and controls in a longitudinal study. ML algorithm empirically classifies individuals based on mathematical calculations of individual’s multimodal MMN or each delta, theta, or alpha activities. We then performed analyses to characterize the cognitive, symptom severity, and functioning performances of the empirically derived clusters to address the question whether data driven classification results were clinically meaningful.

Methods

Participants

Demographics, MMN, cognition (NCCB MATRICS Consensus Cognitive Battery), and clinical data of 33 healthy controls and 20 FEP patients were collected. Eighteen of the FEP patients also had 6-month follow-up data (Table 1). Patients were identified and recruited for the study within 12 months of first episode of psychosis. After enrollment, clinical diagnostic interview and the series of tests including EEG and neurocognitive tests were administered within 45 days. Each subject was assessed by the Structured Clinical Interview for DSM-IV (SCID). Patients were clinically stable. Study inclusion criteria were: 1) age between 18 and 45 years; 2) fluency in English; 3) IQ > 70; 4) patients with FEP diagnosed with SZ, schizoaffective disorder, schizophreniform disorder, psychotic disorder not otherwise specified, or psychotic bipolar disorder. Exclusion criteria consisted of: 1) diagnosed neurological disorder; 2) brain injury including stroke or serious head injury resulting in loss of consciousness; 4) hearing impairments, blindness, or deafness; 5) electroconvulsive therapy within the past 6 months; 6) outside the age range of 18–45 years. HC subjects were recruited from the Partners Research Portal and subject to the same exclusion criteria plus the following: no current or past history of psychotic or affective disorders, no substance abuse or previous chronic dependence, and no first-degree relative with a history of psychosis or bipolar disorder. Patients with substance abuse or dependence within 6 months were excluded. As a history or lifetime diagnosis of substance abuse or dependence is common among patients, FEP patients with previous substance abuse history were not an exclusion. Similarly, depression is common in the general population (~20–25%), healthy controls having relatives with a history of depression were not an exclusion. The study was approved by the McLean Hospital Institutional Review Board. All subjects provided written informed consent after receiving a complete description of the study.

TABLE 1

Table 1 Comparisons between controls, baseline patients, and 6-month follow-up patients.

Neurocognitive, Functioning, and Clinical Assessments

Neurocognition was assessed using the MATRICS Consensus Cognitive Battery [MCCB; (43)]. A MCCB neurocognitive composite score (the average of all MATRICS tasks excluding the social cognition MSCEIT task) was calculated. Social Cognition was assessed using two measures: the Awareness of Social Inference Test (TASIT)—Part Two (44, 45) which measures social inference/Theory of Mind, and the MSCEIT from the MCCB, which measures social and emotional reasoning.

Functional capacity was assessed using the UCSD Performance-Based Skills Assessment, Brief (UPSA-B) (46, 47). The UPSA-B is a performance-based measure designed to evaluate participants’ abilities to perform everyday tasks considered necessary for independent functioning in the community. Total scores range from 0 to 100 points; higher scores reflect better performance. Community functioning was evaluated using an abbreviated version of the Multnomah Community Ability Scale (MCAS) (48), an interview-based measure developed for assessment of community outcomes in psychiatric populations. This brief version probes several aspects of community functioning including independence in daily living, instrumental role functioning, and social interest and engagement (49–51). Clinical assessment was performed using the Positive and Negative Syndrome Scale (PANSS) subscales for Positive, Negative, and General symptoms (52).

EEG Procedures and Data Processing

The electroencephalogram (EEG) was recorded continuously using the BioSemi Active Two system (BioSemi Inc, Amsterdam, Netherlands) at a digitization rate of 512 Hz, with a bandpass of DC–104 Hz, and a Common Mode Sense (CMS) as the reference (PO2 site) using a 64-channel electrode cap. EOG electrodes were placed below and at the outer canthi of the left eye. A duration MMN paradigm was used to elicit MMN. Stimuli consisted of 1,200 trials presented to the subjects through foam insert earphones. 85% of the stimuli were standard [S1] tones (1,000 Hz, 100 ms), and 15% were duration deviant [S2] tones (1,000 Hz, 150 ms) with an inter-stimulus interval of 200 ms. Participants were instructed to watch a silent cartoon/video clip (BBC natural program or Charles Brown) during the stimulus presentation.

Data processing and analysis pipelines are presented in Figure 1 Flow Chart. Data processing was performed offline using Brain Vision Analyzer 2 (Brain Products GmbH, Munich, Germany) and MATLAB R2017b (The MathWorks, Massachusetts, USA) and blind to group membership using automated procedures. Signals were re-referenced to an average of the mastoids and bandpass filtered between 0.01 and 20 Hz. Data were segmented by stimulus marker from −100 to 400 ms for MMN analysis and from −100 to 280 ms for frequency analysis. Segments were baseline corrected using −100 to 0 ms pre-stimulus time and eye-blink corrected using established measures (53). Artifact rejection for individual channels was performed and a given segment was rejected if the voltage gradient exceeded 50 μV/ms, amplitude was +/−100 μV, or the signal was flat (<0.5 μV for >100 ms).

FIGURE 1

Figure 1 Data processing and analysis pipelines.

MMN waveforms were generated by subtracting standard (S1) from the deviant (S2) waveforms and the MMN amplitude was calculated as the mean amplitude (uV) between the time window of 120 to 250 s. Event-related low frequency measures were computed from the single-trial segments. Single-trial S1 and S2 segments were extracted after artifact rejection procedure using the Morlet wavelet transformation (squared wavelet transformation (uV²) for delta: 1–4 Hz; theta: 4–8 Hz; alpha: 7.5–13 Hz). Originally, there were a total of 1,020 S1 and 180 S2 segments. Artifact rejection procedure and an additional step for removing bad intervals led to having approximately 981 S1 segments and 172 S2 segments for each subject per frequency band.

For each frequency band, we computed the sum of the average difference (AverageDifference) index to capture the overall differences between the averaged S2 and S1 segments in each of the 24 channels (see equation below). First we computed the average of S2 (AvgS2) and S1 (AvgS1) in each channel ‘e’. C_S2 is the number of S2 segments and C_S1 is the number of S1 segments (e.g., C_S1 = 981 for S1 segments vs. C_S2 = 172 for S2 segments). Then AverageDifference is computed by summing up the difference between AvgS2 and AvgS1 over time ‘t’, starting from the stimulus onset (0 ms) to the end of the segment (280 ms).

The AverageDifference for each channel e is defined as the following:

$$AverageDifferenc{e}_e={\Sigma }_{t=time}(AvgS{2}_e^{(t)}-AvgS{1}_e^{(t)})$$

where:

$$AvgS1=\frac{{\Sigma }_i^{C_{S1}}S{1}_i}{C_{S1}}$$

$$AvgS2=\frac{{\Sigma }_i^{C_{S2}}S{2}_i}{C_{S2}}$$

Since AverageDifference was derived from squared wavelet transformed values and the summation of differences between AvgS2 and AvgS1, it is susceptible to extreme values which can significantly affect the clustering results. For instance, clustering a sample with one extreme value into two clusters can cause one cluster to contain only one subject while the other cluster contains the rest. To identify extreme outliers, we applied Median Absolute Deviation (MAD) procedure on the sum of the 24 channels, considered any observation with the AverageDifference value over three deviations away from the median to be an “outlier,” and removed such observations from clustering analyses. This procedure removed a total of four observations (two controls and two follow-up FEP patients).

The patterns of event-related responses from both MMN ERP (Figure 2) and averaged S1 and S2 activities (Figure 3) showed greater EEG signals at the frontal and central electrode sites than at the parietal sites, consistent with the literature (17). Therefore, data from 24 frontocentral sites (AFz, AF3, AF4, Fz, F1, F2, F3, F4, F5, F6, FCz, FC1, FC2, FC3, FC4, FC5, FC6, Cz, C1, C2, C3, C4, C5, C6) were used in the subsequent analyses.

FIGURE 2

Figure 2 Scalp voltage topography maps, showing grand average of MMN amplitudes in controls (top), FEP patients at baseline (middle), and FEP patients at 6-month follow-up (bottom), across latency window from −100 to 400 ms. The frontal and central channels showed the strongest MMN responses.

FIGURE 3

Figure 3 Averaged S1 and S2 responses across 40 channels of all subjects. X axis: time (−100 ms to 280ms). Y axis: squared wavelet values. In all participants, the frontal and central channels showed the greater averaged S1 and S2 response patterns than parietal channels.

Cluster Analysis

To stratify subjects into more homogeneous subgroups, each individual’s MMN amplitudes and AverageDifference values across 24 channels were used to derive cluster assignments. The k-means algorithm was used. We applied the elbow method to empirically estimated the optimal number of clusters (54, 55). The elbow method calculates the cost function J for each of the cluster numbers (e.g., from 1 to 10) by minimizing the error. The steeper drop of cost function (error) the better modelling of the data. The empirical elbow method indicates that there were two distinct clusters (see Figure 4 and Supplementary Figures S1 and S2).

FIGURE 4

Figure 4 Top: Grand average of individual’s AverageDifference of delta frequency in healthy Control (Right) and FEP patient baseline (Left) across 24 channels. X axis: electrode sites; Y axis (far left): individual participants; Y axis (far right): the magnitude of the AverageDifference values. Bottom: The elbow method plots. X-axis: number of clusters; Y-axis: error/cost function. The cost (error) value drop is shown for each cluster solution.

Clustering Analyses Using MMN Amplitude Over 24 Channels

Because this study includes controls and a patient cohort with longitudinal follow-up data, four sets of analyses were run, clustering among (i) controls and patients at baseline, (ii) controls and patients at follow-up, (iii) patients at baseline and follow-up, and (iv) patients’ MMN changes over 6-month period (i.e., follow-up MMN amplitude minus baseline MMN amplitude at each electrode channel). Once clusters were determined at each run, the resulting cluster assignments were mapped onto individuals’ clinical, cognitive, and functioning performances, including real-life functioning (UPSA, MCAS), neurocognition, social cognition (TASIT, MSCEIT or MATRICS Social Subscore), and symptom severity (PANSS), in order to characterize each cluster profile (Figure 1).

Clustering Analyses Using AverageDifference Over 24 Channels

Three separate clustering analyses were run based on the AverageDifference index, clustering among (i) controls and patients at baseline (ii) controls and patients at follow-up, and (iii) patients at baseline and patients at follow-up. Once clusters were determined at each run, the resulting cluster assignments were mapped onto individuals’ clinical, cognitive, and functioning performances, as described earlier.

Results

Clustering Analyses Using MMN Amplitude

Clustering results using MMN amplitude over 24 frontocentral channels were shown in Table 2A. In all three sets of clustering analyses (HC & baseline FEP; HC & follow-up FEP; baseline & follow-up FEP), the MMN amplitudes across all 24 electrode sites were larger (i.e., more negative) in Cluster 1 than those in Cluster 2. In each channel differences between cluster 1 and 2 were significant at p < 0.05.

TABLE 2

Table 2 Clustering results of MMN amplitude and the clinical, cognitive, and functional characterizations of clusters.

We labeled the individuals in Cluster 1 as “Better functioning” and Cluster 2 as “Poorer functioning”, respectively. The demographic, clinical, cognitive, functioning profiles of these clusters are presented in Table 2B. Patient only clustering results were in Supplementary Table S1. Results of clustering among controls and patient baseline (Table 2B, left, Supplementary Figure S3) showed individuals in the “Better functioning” group, 31% of whom were patients and performed better but not significantly on all of the neurocognition, social cognition, and functioning measures, with an exception of UPSA task. Results of clustering among controls and follow-up patients (Table 2B, middle, Supplementary Figure S4) showed individuals in the “Better functioning” group, 30% of whom were patients and performed significantly better on all except two of the neurocognition, social cognition, and functioning measures. Results of clustering among baseline and follow-up patients (Table 2B, right) showed that patients in the “Better functioning” performed better on all the neurocognition, social cognition, and functioning measures, as well as had lower symptom severity scores (less symptomatic), and that group differences were significant on MCAS independent, MCAS social subscore, and MATRICS social subscore. 30% of patients in the “Better functioning” had SZ diagnosis.

Clustering Analysis Using Changes in MMN Amplitude Over 6 Months

Results of patient’s MMN change over 6-months across 24 frontocentral channels were shown in Table 3A. Consistently across all 24 channels, Cluster 1 patients had, on average, bigger MMN amplitudes at follow-up than their baseline MMN, resulting in more negative MMN changes. In contrast, patients in the Cluster 2 group had smaller MMN at follow-up than their baseline amplitude, resulting in more positive MMN change. In each channel except FC5 site, differences between clusters 1 and 2 were significant at p < 0.05 (Table 3A). These results indicated that patients in Cluster 1 as a group had improved MMN over time whereas patients in Cluster 2 had worsening MMN responses. The demographic, clinical, cognitive, functioning profiles of “better” and “poorer” clusters are presented in Table 3B. Patients in the “Better functioning” cluster performed better on all the neurocognition, social cognition, and functioning measures, as well as had lower symptom severity scores than those in the “Poorer functioning” group. However, differences in each cognitive or functioning measure were not statistically significant except for the MCAS social subscore.

TABLE 3

Table 3 Clustering results of MMN change over 6-months and the clinical, cognitive, and functional characterizations of clusters.

The Magnitude of Delta, Theta, Alpha Frequency Activity in Controls and Patients

The overall magnitudes of the AverageDifference indices across three frequency bands were presented in Figure 5. Results showed that in both controls and patients the magnitudes of AverageDifference at delta frequency were statistically significantly higher than theta or alpha frequencies across most of the 24 channels (Figure 5). In controls p-value differences between delta and theta and between delta and alpha were p =1.1E-28 and p = 1.8E-22, respectively; in patient’s baseline, p-value differences were p =1.7E-21 and p = 1.1E-05, respectively. Thus, clustering analyses were only performed using AverageDifference index on delta frequency.

FIGURE 5

Figure 5 Grand average of AverageDifference across Delta, Theta and Alpha frequency bands in HC (bottom three rows), FEP baseline (middle three rows), and FEP follow-up (top three rows). X axis: electrode sites; Y axis (far right): the absolute value of the average AverageDifference. In all groups, the delta frequency showed the strongest signal across all 24 channels.

Clustering Analysis Using AverageDifference on Delta Frequency Measure

Table 4A showed results using AverageDifference index on delta frequency over 24 frontocentral channels. In all three sets of clustering analyses (HC & baseline FEP; HC & follow-up FEP; baseline & follow-up FEP), the AverageDifference values across all 24 channels were greater in Cluster 1 than in Cluster 2, indicating that individuals as a group in Cluster 1 had AvgS2 > AvgS1 response, resulting in positive AverageDifference values, whereas individuals in Cluster 2 had AvgS2 < AvgS1 values, resulting in negative AverageDifference.

TABLE 4

Table 4 Clustering results of AverageDifference index and the clinical, cognitive, and functional characterizations of clusters.

We labeled Cluster 1 as “higher AverageDifference” and Cluster 2 as “lower AverageDifference”, respectively. The demographic, clinical, cognitive, functioning profiles of these clusters were presented in Table 4B. Patient only clustering results were in Supplementary Table S2. Results of the clustering among controls and baseline patients (Table 4B, left) showed that 41% of individuals in the “higher AverageDifference” group were patients. Individuals in the “higher AverageDifference” performed better on MCAS functioning measures (total score, independent subscore, and social subscore) than those in the “lower AverageDifference” but not on neurocognition, social cognition, or UPSA functioning tasks. Among controls and follow-up patients (Table 4B, middle), results showed that 29% in the “higher AverageDifference” cluster were patients. Individuals in the “higher AverageDifference” performed better on MCAS independent subscore and MATRICS social subscore, but worse on neurocognition, TASIT, and other functioning measures. Results of clustering among baseline and follow-up patients (Table 4B, right) showed patients in the “higher AverageDifference” cluster performed better on MCAS measures (total score, independent subscore, and social subscore) and had lower symptom severity compared to those in the “lower AverageDifference” group but not on all the other measures (UPSA, neurocognition, MATRICS social, and TASIT). None was statistically significant at p < 0.05.

Discussion

In this study, we used an unsupervised ML k-means algorithm to empirically stratify individuals into more homogeneous subgroups on the basis of multi-channel MMN data in a sample of controls and patients with FEP. We then characterized the functional, cognitive, and clinical profiles of these neurobiologically derived subgroups. We found, firstly, a consistent pattern of two distinctive subgroups across 24 frontocentral channels. Secondly, the two subgroups derived from MMN amplitude could be mapped onto either better or poorer clinical, cognitive, and functioning profiles. Thirdly, we examined the longitudinal MMN change over time and identified two subgroups of patients, one who showed improved MMN overtime and one who showed worsening of MMN overtime. Patients with improved MMN also had better follow-up clinical, cognitive, and functioning profile than those with worsening MMN. Fourthly, among the low frequency in the delta, theta, and alpha frequency bands, delta frequency appeared to be mostly relevant to the observed S1 and S2 EEG responses in both controls and patients. Finally, although delta frequency AverageDifference index could also empirically produce two distinctive subgroups, individuals in the higher AverageDifference cluster were not necessarily associated with better clinical, cognitive, and functioning profile than those in the lower AverageDifference group. To our knowledge, our study is the first to link neurophysiological derived subgroups with distinct cognitive and functional outcome characteristics. In addition, we demonstrated that variability in MMN change overtime is associated with symptom and functional outcomes.

Heterogeneity in Patients and Controls

Heterogeneity is a major barrier to understand disease mechanisms and identify individuals with different recovery trajectories. As Figure 4 shows, the EEG data of our sample clearly indicates that there exists a large variability not only among patients but also among controls. In addition, when both controls and patients were included in the MMN clustering analyses, 30% of patients were classified in the “better functioning” group. The combination of clustering techniques and multi-channel MMN activity employed in the current study facilitates the identification of neurobiological homogeneous subgroups. In two prior studies, we have used K-means analyses and identified distinct “Bio-classes” (38) among patients and controls and unique functional trajectories (4) among FEP patients that do not respect clinical diagnosis boundaries. Within each class, individuals shared a similar neurobiological profile or functional outcome trajectories that uniquely distinguished among the groups. These studies present a diagnosis-free approach to integrate information across biomarkers, yielding neurobiologically distinct subgroups and providing strong evidence supporting the superiority of neurobiological vs. clinical classification in differentiating psychotic disorders.

The presence of distinct neurobiological profiles among controls and patients brings into question the appropriateness of using diagnosis based on patient–control comparison analysis in MMN research, particularly during the early stage of illness. The relatively small effect size of dMMN deficit (20) reported in the meta-analyses is consistent with the notion of a large variability among FEP patients. The two subgroups identified in each of the three clustering models using 24-channel MMN are highly consistent and concordant in terms of the overall sample characteristics. MMN across 24 channels was consistently and significantly bigger in one subgroup over the other (Table 2A). We labeled the larger MMN subgroup as “Better functioning” as individuals in this group were consistently showing an overall pattern of better cognitive, social cognition, and functioning performances than the “poor functioning” individuals, regardless of whether controls were included in the models or not. That is, although significant differences were not observed in some clinical, cognitive, and functioning variables, the overall “gestalt” pattern was consistent. These results suggested that the ML data driven approach is a useful strategy to reduce heterogeneity and provide insight into clinically meaningful subgroups of a cohort. Also, results among baseline and follow-up patient clustering (Table 2B, right) showed that, patients in the “Better functioning” performed better on all the neurocognition, social cognition, and functioning measures, as well as had lower symptom severity scores (less symptomatic). Among patients in the “Better functioning”, 30% had SZ diagnosis. This result is in line with the literature that patients with SZ generally exhibit more impairment and greater symptom severity compared to patients with affective psychosis. One prior study (35) has found that the most impaired baseline MMN amplitudes corresponded to the most severe functional disability at follow-up, consistent with the findings obtain in our study.

Longitudinal MMN Change Is Associated With Better Clinical and Outcome Characteristics

Results of our analyses showed that MMN changes across 24 channels were significantly improved in one subgroup over the other and that patients with improved MMN over the 6-months follow-up period also had better follow-up clinical, cognitive, and functioning profile than those with worsening MMN (Tables 3A, B). It is striking that we observed again a consistent overall “gestalt” pattern that patients in the “Better functioning” cluster performed better on all the neurocognition, social cognition, and functioning measures, as well as had lower symptom severity scores than those in the “Poorer functioning” group. Although differences in most cognitive and functioning variables were largely non-significant, this is likely due to small sample size in each of the clusters (Ns = 10 and 8). Nonetheless, the overall pattern is consistent and among the variables examined, the UPSA, MATRICS social, and PANSS total had the biggest mean differences. These results support the notion that the ML data driven approach is useful to explore heterogeneity and facilitate the identification of neurobiological homogeneous subgroups. Koshiyama et al. (56) reported that MMN of FEP patients do not change significantly over time, while (36) observed that MMN deteriorated in patients over time. These prior reports also indicate heterogeneity in MMN change over time among FEP patients. Our present findings suggest that combining ML and follow-up clinical characterization approaches can potentially identify individuals at greater risk of poorer functional outcomes at a “critical period” of neuronal and psychosocial plasticity and for whom there is a “window of opportunity” for treatment to achieve disproportionately favorable outcomes. These individuals can be targeted for earlier, more aggressive treatment interventions, both pharmacologically and psychosocial/cognitive intervention therapy, to reduce function deterioration and improve recovery.

Significant Contribution of Delta Frequency During MMN

In this study, we explored the underlying EEG frequency responses during MMN by decomposing MMN data into low frequency range activity and calculated frequency specific “sum difference” (AverageDifference) indices to capture the overall differences between the averaged S2 and S1 segments. We found that the magnitudes of AverageDifference at delta frequency were significantly higher than those of theta or alpha frequencies across all 24 channels in controls and patients (Figure 5), suggesting that AverageDifference of delta frequency plays an important role in the MMN generation. Our results are consistent with Hong et al., 2012 who observed that delta range activities were found to explain the most variance of observed MMN abnormalities in SZ (41).

Comparison of MMN and AverageDifference of Delta Frequency

As shown in Table 4, two distinct subgroups could also be consistently obtained in all three clustering models using the AverageDifference index of delta frequency, one cluster with “higher AverageDifference” values and the other with “lower AverageDifference”. While K-means generated a consistent pattern of two clusters using either multimodal MMN or low frequency measure, individuals in the higher AverageDifference cluster were not necessarily associated with better clinical, cognitive and functioning profile than those in the lower AverageDifference group. There are two major differences between MMN and the AverageDifference of delta frequency. MMN waveforms were generated directly by subtracting the average of S1 from the average of S2 waveforms between the time window of 120 to 250 ms. Although the same window was used for the AverageDifference, squared wavelet transformation is applied to each single trial response of the original S1 and S2 signals before computing the sum of the difference between averaged S1 and S2. In addition, AverageDifference index was derived using delta frequency only, which constitutes a subset of total MMN signals. As a result, the associated clinical and functioning variances were not fully captured in the AverageDifference of delta frequency. Our results suggest that MMN appeared to yield clinically meaningful interpretation, and MMN is superior to the AverageDifference index as a neurobiological measure for identifying clinically distinct subgroups and the application of squared wavelet transformation may not be an optimal method.

Our study has a number of limitations. Although several interesting and consistent findings were found, the study sample size is relatively small, particularly the patient’s longitudinal data. K-means is an exploratory research tool to discover new patterns. Although k-means clustering can sufficiently uncover patterns with a relatively small number of subjects (57, 58), supervised learning methods would be used, and a larger number of subjects in the future is needed to prove the results are generalizable for a larger population (59). Second, there are no follow-up data for controls, which limit our ability to determine the stability of cognitive, functioning, and EEG measures or typical changes that occur in all individuals. Third, because our focus was on deriving more homogeneous subgroups using agnostic approach on the basis of MMN or low frequency activities, we did not investigate the degree of consistence within patients by comparing PANSS-derived subgroups with MMN derived subgroups.

Conclusions

In conclusion, the ML data-driven approach is a useful tool in FEP psychosis research to address heterogeneity and facilitate identification of clinically meaningful subgroups and patterns between MMN and clinical, cognitive, functioning characteristics.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

The studies involving human participants were reviewed and approved by Partners Human Research Committee. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

XQ, TH, and M-HH contributed to the conception and design of the study. AH organized the database. XQ, SL, and JT performed the statistical analysis. XQ wrote the first draft of the manuscript. M-HH, SL, and JT wrote sections of the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This publication was supported by funds received from National Institute of Mental Health grants NIMH [R01MH109687]: M-HH, PI.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2020.541659/full#supplementary-material

References

1. Waghorn G, Saha S, Harvey C, Morgan VA, Waterreus A, Bush R, et al. ‘Earning and learning’ in those with psychotic disorders: the second Australian national survey of psychosis. Aust New Z J Psychiatry (2012) 46(8):774–85. doi: 10.1177/0004867412452015

CrossRef Full Text | Google Scholar

2. Whiteford HA, Ferrari AJ, Degenhardt L, Feigin V, Vos T. The global burden of mental, neurological and substance use disorders: an analysis from the Global Burden of Disease Study 2010. PloS One (2015) 10(2):e0116820. doi: 10.1371/journal.pone.0116820

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Keshavan MS, Clementz BA, Pearlson GD, Sweeney JA, Tamminga CA. Reimagining psychoses: an agnostic approach to diagnosis. Schizophr Res (2013) 146(1-3):10–6. doi: 10.1016/j.schres.2013.02.022

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Hall MH, Holton KM, Ongur D, Montrose D, Keshavan MS. Longitudinal trajectory of early functional recovery in patients with first episode psychosis. Schizophr Res (2019) 209:234–44. doi: 10.1016/j.schres.2019.02.003

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Allott K, Liu P, Proffitt TM, Killackey E. Cognition at illness onset as a predictor of later functional outcome in early psychosis: systematic review and methodological critique. Schizophr Res (2011) 125(2-3):221–35. doi: 10.1016/j.schres.2010.11.001

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Menezes NM, Arenovich T, Zipursky RB. A systematic review of longitudinal outcome studies of first-episode psychosis. Psychol Med (2006) 36(10):1349–62. doi: 10.1017/S0033291706007951

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Naatanen R, Escera C. Mismatch negativity: clinical and other applications. Audiol Neurootol (2000) 5(3-4):105–10. doi: 10.1159/000013874

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Light GA, Naatanen R. Mismatch negativity is a breakthrough biomarker for understanding and treating psychotic disorders. Proc Natl Acad Sci USA (2013) 110(38):15175–6. doi: 10.1073/pnas.1313287110

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Grimm S, Escera C, Nelken I. Early indices of deviance detection in humans and animal models. Biol Psychol (2016) 116:23–7. doi: 10.1016/j.biopsycho.2015.11.017

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Alho K, Escera C, Schröger E. Event-Related Brain Potential Indices of Involuntary Attention to Auditory Stimulus Changes. In: Polich J, editor. Detection of Change: Event-Related Potential and fMRI Findings. Boston, MA: Springer US (2003). p. 23–40.

Google Scholar

11. Naatanen R, Kahkonen S. Central auditory dysfunction in schizophrenia as revealed by the mismatch negativity (MMN) and its magnetic equivalent MMNm: a review. Int J Neuropsychopharmacol (2009) 12(1):125–35. doi: 10.1017/S1461145708009322

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Rissling AJ, Park SH, Young JW, Rissling MB, Sugar CA, Sprock J, et al. Demand and modality of directed attention modulate “pre-attentive” sensory processes in schizophrenia patients and nonpsychiatric controls. Schizophr Res (2013) 146(1-3):326–35. doi: 10.1016/j.schres.2013.01.035

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Avissar M, Xie S, Vail B, Lopez-Calderon J, Wang Y, Javitt DC. Meta-analysis of mismatch negativity to simple versus complex deviants in schizophrenia. Schizophr Res (2018) 191:25–34. doi: 10.1016/j.schres.2017.07.009

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Rissling AJ, Miyakoshi M, Sugar CA, Braff DL, Makeig S, Light GA. Cortical substrates and functional correlates of auditory deviance processing deficits in schizophrenia. NeuroImage Clin (2014) 6:424–37. doi: 10.1016/j.nicl.2014.09.006

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Uno Y, Coyle JT. Glutamate hypothesis in schizophrenia. Psychiatry Clin Neurosci (2019) 73(5):204–15. doi: 10.1111/pcn.12823

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Edgar JC. Identifying electrophysiological markers of autism spectrum disorder and schizophrenia against a backdrop of normal brain development. Psychiatry Clin Neurosci (2020) 74(1):1–11. doi: 10.1111/pcn.12927

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Wynn JK, Sugar C, Horan WP, Kern R, Green MF. Mismatch negativity, social cognition, and functioning in schizophrenia patients. Biol Psychiatry (2010) 67(10):940–7. doi: 10.1016/j.biopsych.2009.11.024

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Rissling AJ, Braff DL, Swerdlow NR, Hellemann G, Rassovsky Y, Sprock J, et al. Disentangling early sensory information processing deficits in schizophrenia. Clin Neurophysiol (2012) 123(10):1942–9. doi: 10.1016/j.clinph.2012.02.079

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Hall M-H, Rijsdijk FV, Picchioni M, Schulze K, Ettinger U, Toulopoulou T, et al. Substantial Shared Genetic Influences on Schizophrenia and Event-Related Potentials. Am J Psychiatry (2007) 164:804–12. doi: 10.1176/appi.ajp.164.5.804

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Haigh SM, Coffman BA, Salisbury DF. Mismatch Negativity in First-Episode Schizophrenia: A Meta-Analysis. Clin EEG Neurosci (2017) 48(1):3–10. doi: 10.1177/1550059416645980

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Javitt DC, Spencer KM, Thaker GK, Winterer G, Hajos M. Neurophysiological biomarkers for drug development in schizophrenia. Nat Rev Drug Discovery (2008) 7(1):68–83. doi: 10.1038/nrd2463

CrossRef Full Text | Google Scholar

22. Koshiyama D, Kirihara K, Tada M, Nagai T, Fujioka M, Usui K, et al. Reduced Auditory Mismatch Negativity Reflects Impaired Deviance Detection in Schizophrenia. Schizophr Bull (2020) 46(4):937–46. doi: 10.1093/schbul/sbaa006

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Ehrlichman RS, Maxwell CR, Majumdar S, Siegel SJ. Deviance-elicited changes in event-related potentials are attenuated by ketamine in mice. J Cogn Neurosci (2008) 20(8):1403–14. doi: 10.1162/jocn.2008.20097

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Heresco-Levy U, Javitt DC. The role of N-methyl-D-aspartate (NMDA) receptor-mediated neurotransmission in the pathophysiology and therapeutics of psychiatric syndromes. Eur Neuropsychopharmacol J Eur Coll Neuropsychopharmacol (1998) 8(2):141–52. doi: 10.1016/S0924-977X(97)00050-3

CrossRef Full Text | Google Scholar

25. Gil-da-Costa R, Stoner GR, Fung R, Albright TD. Nonhuman primate model of schizophrenia using a noninvasive EEG method. Proc Natl Acad Sci U.S.A. (2013) 110(38):15425–30. doi: 10.1073/pnas.1312264110

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Preskorn SH, Gawryl M, Dgetluck N, Palfreyman M, Bauer LO, Hilt DC. Normalizing effects of EVP-6124, an alpha-7 nicotinic partial agonist, on event-related potentials and cognition: a proof of concept, randomized trial in patients with schizophrenia. J Psychiatr Pract (2014) 20(1):12–24. doi: 10.1097/01.pra.0000442935.15833.c5

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Naatanen R, Kujala T, Kreegipuu K, Carlson S, Escera C, Baldeweg T, et al. The mismatch negativity: an index of cognitive decline in neuropsychiatric and neurological diseases and in ageing. Brain J Neurol (2011) 134(Pt 12):3435–53. doi: 10.1093/brain/awr064

CrossRef Full Text | Google Scholar

28. Baldeweg T, Klugman A, Gruzelier J, Hirsch SR. Mismatch negativity potentials and cognitive impairment in schizophrenia. Schizophr Res (2004) 69(2-3):203–17. doi: 10.1016/j.schres.2003.09.009

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Light GA, Swerdlow NR, Braff DL. Preattentive sensory processing as indexed by the MMN and P3a brain responses is associated with cognitive and psychosocial functioning in healthy adults. J Cogn Neurosci (2007) 19(10):1624–32. doi: 10.1162/jocn.2007.19.10.1624

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Rasser PE, Schall U, Todd J, Michie PT, Ward PB, Johnston P, et al. Gray matter deficits, mismatch negativity, and outcomes in schizophrenia. Schizophr Bull (2011) 37(1):131–40. doi: 10.1093/schbul/sbp060

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Kawakubo Y, Kamio S, Nose T, Iwanami A, Nakagome K, Fukuda M, et al. Phonetic mismatch negativity predicts social skills acquisition in schizophrenia. Psychiatry Res (2007) 152(2-3):261–5. doi: 10.1016/j.psychres.2006.02.010

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Light GA, Braff DL. Mismatch negativity deficits are associated with poor functioning in schizophrenia patients. Arch Gen Psychiatry (2005) 62(2):127–36. doi: 10.1001/archpsyc.62.2.127

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Erickson MA, Ruffle A. Gold JM. A Meta-Analysis of Mismatch Negativity in Schizophrenia: From Clinical Risk to Disease Specificity and Progression. Biol Psychiatry (2016) 79(12):980–7. doi: 10.1016/j.biopsych.2015.08.025

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Umbricht D, Krljes S. Mismatch negativity in schizophrenia: a meta-analysis. Schizophr Res (2005) 76(1):1–23. doi: 10.1016/j.schres.2004.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Kaur M, Lagopoulos J, Lee RS, Ward PB, Naismith SL, Hickie IB, et al. Longitudinal associations between mismatch negativity and disability in early schizophrenia- and affective-spectrum disorders. Prog Neuropsychopharmacol Biol Psychiatry (2013) 46:161–9. doi: 10.1016/j.pnpbp.2013.07.002

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Salisbury DF, Kuroki N, Kasai K, Shenton ME, McCarley RW. Progressive and interrelated functional and structural evidence of post-onset brain reduction in schizophrenia. Arch Gen Psychiatry (2007) 64(5):521–9. doi: 10.1001/archpsyc.64.5.521

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Marquand AF, Wolfers T, Mennes M, Buitelaar J, Beckmann CF. Beyond Lumping and Splitting: A Review of Computational Approaches for Stratifying Psychiatric Disorders. Biol Psychiatry Cognit Neurosci Neuroimaging (2016) 1(5):433–47. doi: 10.1016/j.bpsc.2016.04.002

CrossRef Full Text | Google Scholar

38. Hall MH, Smoller JW, Cook NR, Schulze K, Hyoun Lee P, Taylor G, et al. Patterns of deficits in brain function in bipolar disorder and schizophrenia: a cluster analytic study. Psychiatry Res (2012) 200(2-3):272–80. doi: 10.1016/j.psychres.2012.07.052

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Hsiao FJ, Wu ZA, Ho LT, Lin YY. Theta oscillation during auditory change detection: An MEG study. Biol Psychol (2009) 81(1):58–66. doi: 10.1016/j.biopsycho.2009.01.007

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Fuentemilla L, Marco-Pallares J, Munte TF, Grau C. Theta EEG oscillatory activity and auditory change detection. Brain Res (2008) 1220:93–101. doi: 10.1016/j.brainres.2007.07.079

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Hong LE, Moran LV, Du X, O’Donnell P, Summerfelt A. Mismatch negativity and low frequency oscillations in schizophrenia families. Clin Neurophysiol (2012) 123(10):1980–8. doi: 10.1016/j.clinph.2012.03.011

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Javitt DC, Lee M, Kantrowitz JT, Martinez A. Mismatch negativity as a biomarker of theta band oscillatory dysfunction in schizophrenia. Schizophr Res (2018) 191:51–60. doi: 10.1016/j.schres.2017.06.023

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Marder SR, Fenton W. Measurement and Treatment Research to Improve Cognition in Schizophrenia: NIMH MATRICS initiative to support the development of agents for improving cognition in schizophrenia. Schizophr Res (2004) 72(1):5–9. doi: 10.1016/j.schres.2004.09.010

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Kern RS, Green MF, Fiske AP, Kee KS, Lee J, Sergi MJ, et al. Theory of mind deficits for processing counterfactual information in persons with chronic schizophrenia. Psychol Med (2009) 39(4):645–54. doi: 10.1017/S0033291708003966

PubMed Abstract | CrossRef Full Text | Google Scholar

45. McDonald S, Flanagan S, Rollins J, Kinch J. TASIT: A new clinical tool for assessing social perception after traumatic brain injury. J Head Trauma Rehabil (2003) 18(3):219–38. doi: 10.1097/00001199-200305000-00001

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Mausbach BT, Harvey PD, Goldman SR, Jeste DV, Patterson TL. Development of a brief scale of everyday functioning in persons with serious mental illness. Schizophr Bull (2007) 33(6):1364–72. doi: 10.1093/schbul/sbm014

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Mausbach BT, Harvey PD, Pulver AE, Depp CA, Wolyniec PS, Thornquist MH, et al. Relationship of the Brief UCSD Performance-based Skills Assessment (UPSA-B) to multiple indicators of functioning in people with schizophrenia and bipolar disorder. Bipolar Disord (2010) 12(1):45–55. doi: 10.1111/j.1399-5618.2009.00787.x

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Barker S, Barron N, McFarland BH, Bigelow DA, Carnahan T. A community ability scale for chronically mentally ill consumers: Part II. Applications. Community Ment Health J (1994) 30(5):459–72. doi: 10.1007/BF02189063

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Monaghan CK, Brickman S, Huynh P, Ongur D, Hall MH. A longitudinal study of event related potentials and correlations with psychosocial functioning and clinical features in first episode psychosis patients. Int J Psychophysiol Off J Int Organ Psychophysiol (2019) 145:48–56. doi: 10.1016/j.ijpsycho.2019.05.007

CrossRef Full Text | Google Scholar

50. Zhou TH, Mueller NE, Spencer KM, Mallya SG, Lewandowski KE, Norris LA, et al. Auditory steady state response deficits are associated with symptom severity and poor functioning in patients with psychotic disorder. Schizophr Res (2018) 201:278–86. doi: 10.1016/j.schres.2018.05.027

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Lewandowski KE, Cohen BM, Keshavan MS, Sperry SH, Ongur D. Neuropsychological functioning predicts community outcomes in affective and non-affective psychoses: a 6-month follow-up. Schizophr Res (2013) 148(1-3):34–7. doi: 10.1016/j.schres.2013.05.012

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull (1987) 13(2):261–76. doi: 10.1093/schbul/13.2.261

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Gratton G, Coles MG, Donchin E. A new method for off-line removal of ocular artifact. Electroencephalogr Clin Neurophysiol (1983) 55(4):468–84. doi: 10.1016/0013-4694(83)90135-9

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Kodinariya TM, Makwana PR. Review on determining number of Cluster in K-Means Clustering. Int J Advance Res Comput Sci Manage Stud (2013) 1(6):90–5.

Google Scholar

55. Ng AY, Coates A. Learning Feature Representations with K-means. In: Montavon G, Orr GB, Müller K-R, editors. Neural Networks: Tricks of the Trade. Berlin, Heidelberg: Springer (2012). p. 561–80.

Google Scholar

56. Koshiyama D, Kirihara K, Tada M, Nagai T, Koike S, Suga M, et al. Duration and frequency mismatch negativity shows no progressive reduction in early stages of psychosis. Schizophr Res (2017) 190:32–8. doi: 10.1016/j.schres.2017.03.015

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Jain AK, Murty MN, Flynn PJ. Data clustering: a review. ACM Comput Surveys (1999) 31(3):264–323. doi: 10.1145/331499.331504

CrossRef Full Text | Google Scholar

58. Jain AK. Data clustering: 50 years beyond K-means. Pattern Recognit Lett (2010) 31:651–66. doi: 10.1016/j.patrec.2009.09.011

CrossRef Full Text | Google Scholar

59. Vabalas A, Gowen E, Poliakoff E, Casson AJ. Machine learning algorithm validation with a limited sample size. PloS One (2019) 14(11):e0224365. doi: 10.1371/journal.pone.0224365

PubMed Abstract | CrossRef Full Text | Google Scholar