Cognitive Impairments in Schizophrenia: A Study in a Large Clinical Sample Using Natural Language Processing

Background: Cognitive impairments are a neglected aspect of schizophrenia despite being a major factor of poor functional outcome. They are usually measured using various rating scales, however, these necessitate trained practitioners and are rarely routinely applied in clinical settings. Recent advances in natural language processing techniques allow us to extract such information from unstructured portions of text at a large scale and in a cost effective manner. We aimed to identify cognitive problems in the clinical records of a large sample of patients with schizophrenia, and assess their association with clinical outcomes.

Methods: We developed a natural language processing based application identifying cognitive dysfunctions from the free text of medical records, and assessed its performance against a rating scale widely used in the United Kingdom, the cognitive component of the Health of the Nation Outcome Scales (HoNOS). Furthermore, we analyzed cognitive trajectories over the course of patient treatment, and evaluated their relationship with various socio-demographic factors and clinical outcomes.

Results: We found a high prevalence of cognitive impairments in patients with schizophrenia, and a strong correlation with several socio-demographic factors (gender, education, ethnicity, marital status, and employment) as well as adverse clinical outcomes. Results obtained from the free text were broadly in line with those obtained using the HoNOS subscale, and shed light on additional associations, notably related to attention and social impairments for patients with higher education.

Conclusions: Our findings demonstrate that cognitive problems are common in patients with schizophrenia, can be reliably extracted from clinical records using natural language processing, and are associated with adverse clinical outcomes. Harvesting the free text from medical records provides a larger coverage in contrast to neurocognitive batteries or rating scales, and access to additional socio-demographic and clinical variables. Text mining tools can therefore facilitate large scale patient screening and early symptoms detection, and ultimately help inform clinical decisions.

Introduction

Despite the relatively limited prevalence of schizophrenia (Sz) worldwide (around 1% of the population), it is one of the largest contributors to the global burden of disease, ranking in the top 10 illnesses (1). Schizophrenia is associated with poor lifestyle (e.g., obesity, smoking) and high suicide rates (2), leading to an estimated reduction in life expectancy by 10–25 years (3). Additionally, Sz is a major cause of disability, ranked third after blindness and paraplegia, resulting in a significant economic and human cost (4).

In particular, cognitive dysfunctions have been associated with poor functional outcomes, and are present in most patients, impairing their ability to establish social interactions, plan, adapt, or solve problems (5). Cognitive dysfunctions in Sz have been extensively studied and documented (6, 7) and typically include impairments in memory, attention, executive functioning, learning, and to a lesser extent social cognition (8).

Over the last decade, several neurocognitive assessment batteries such as the MATRICS Consensus Cognitive Battery (MCCB) (9) or the Cambridge Neuropsychological Test Automated Battery (CANTAB) (10) have been increasingly utilized in the United Kingdom (UK) as outcome measures for clinical trials targeting cognition in schizophrenia (Sz) (11).

Whilst such neuropsychological tests are valuable tools, they remain relatively expensive, may require several hours to administer, and are not routinely applied in clinical practice (12, 13).

Conversely, natural language processing (NLP) allows the harvesting of rich clinical data recorded in the free-text narratives of Electronic Health Records (EHRs), therefore offering an alternative method to derive detailed clinical phenotyping on large volumes of clinical data. NLP has thus far been used to process and analyse information from the unstructured text of EHRs for a variety of healthcare applications, including patients identification for trials, de-identification of records, extraction of disease risk factors or symptoms, or care and cost evaluation (14). Looking more specifically at symptom classification, a recent systematic literature review identified 14 studies presenting symptom information extraction as a primary measure (15), including adverse drug events, depressed mood and mood instability for mental healthcare (16, 17). Several additional studies targeting Sz were also identified: (18–20) detected, respectively, mentions of negative symptoms, poor insight and atypical hallucinations (21), identified abnormal motor behavior, delusion, disorganized thinking, hallucination and negative symptoms. However, to the best of our knowledge, no previous work has focused on extracting cognitive impairments from the unstructured portion of medical records.

In this study, we present a novel NLP tool that identifies cognitive impairments for patients with Sz from the unstructured portions of EHRs. We demonstrate the benefit of using an NLP approach by evaluating it against a routinely administered rating scale, and using the information extracted from free text to explore the association between cognitive problems, socio-demographic factors and clinical outcomes.

Materials and Methods

All the data used in this study was extracted from the South London (UK) and Maudsley NHS Foundation Trust Biomedical Research Center Case Register (CRIS) comprising pseudonymized EHRs of over 350,000 patients (22, 23).

Cognitive Impairments Definition

In order to define the scope of cognitive impairments, we reviewed the domains covered by selected major cognitive test batteries and scales that were specifically developed for or adapted to Sz, including MCCB (9), CANTAB (10), The Brief Assessment of Cognition in Schizophrenia (BACS) (13), the Schizophrenia Cognition Rating Scale (SCoRS) (24), and the Subjective Scale to Investigate Cognition in Schizophrenia (SSTICS) (25).

Several consistent major domains of impairments emerged from this analysis, namely: attention, memory, executive functioning, and social cognition, which were then used to implement the NLP framework.

Measurement Development

We used NLP to identify mentions of cognitive impairments in the unstructured portion of clinical records.

Annotations Dataset

A set of keywords relevant to cognition, spanning the different domains identified (attention, memory, executive functioning, social cognition, and general cognitive impairments) was derived manually from SNOMED-CT (26) and the Unified Medical Language System (27) terminologies, and further enriched with clinicians' feedback.

Fourteen thousand sentences containing these keywords were randomly sampled from documents of patients diagnosed with Sz and active in CRIS between 2007 and 2020. The sentences were then annotated by two medical students for the presence of symptoms (indicating that the impairment is affirmed and relates to the patient, e.g., “ZZZ shows poor concentration,” as opposed to negated or irrelevant mention, e.g., “ZZZ shows good concentration” or “dose adjusted to a serum-lithium concentration of 0.4–1 mmol/L”). The annotations were conducted in a single phase using detailed guidelines designed in conjunction with researchers and clinicians, achieving an overall inter-annotator agreement of 93.7%.

Classification Model

Following the comparative analysis of algorithms for biomedical text classification tasks conducted in Mascio et al. (28), we fine-tuned a transformers-based model, BioBERT (29, 30) using the annotated dataset (code available on github).

BioBERT is a transformers-based model pre-trained on large scale biomedical corpora (31), and consequently already captures medical terminologies and jargon. In order to fine-tune BioBERT to the cognitive symptoms classification task, we used the pre-trained model, added an untrained softmax classification layer at the end, and trained the new model for our task (as illustrated in Figure 1 below) (32).

FIGURE 1

Figure 1. BioBERT fine-tuning.

We used F1 score, precision and recall to estimate the model's performance (33). Precision here measures the number of positive predictions that actually correspond to positive examples [True Positives/(True Positives + False Positives)], and recall measures the number of positive examples correctly predicted as positive [True Positives/(True Positives + False Negatives)]. F1 score balances both precision and recall in one single metric: F1 = 2 × precision × recall/(precision + recall).

Cognitive Scores

The symptoms extracted via the fine-tuned NLP model were used to derive a composite score of cognitive impairment (“NLP-CI”), in the following way:

1. For a given patient at a given point in time, a score of 1 is assigned if a cognitive problem is recorded in any document with that date. This process is repeated for each of the five individual cognitive domains (attention, memory, executive functioning, social cognition, and general cognitive problem), generating a set of five binary scores.

2. The NLP-CI is computed by summing the scores across all individual domains, resulting in a rating scale between 0 and 5, and denoting the number of cognitive symptoms exhibited by the patients at a given date. We also defined its associated binary variable on the basis of a score of 1 or more (“NLP-CI-b”).

A patient will then be assigned, for each date at which documents were recorded, five binary scores indicating the presence or absence of individual cognitive problems, as well as two composite scores indicating the number of different symptoms recorded and the presence of more than one symptom.

Comparison With HoNOS

The Health of the Nation Outcome Scales (HoNOS) has been developed to assess the health and functioning of working age adults with severe mental illness (34), and is routinely administered in most mental health services in the UK (35). HoNOS comprises 12 scales and, of specific interest to our study, a cognitive impairments subscale (“HoNOS-CI”) which aims at measuring cognitive problems severity. This subscale ranges from 0 to 4, 0 meaning no problem, 1,2,3,4 indicating minor, mild, moderate, and severe problems (34).

Despite the fact that the NLP-CI score evaluates the numbers of cognitive domains impaired (as opposed to measuring general cognitive impairment severity) and coverage differences (HoNOS is not administered to children, and does not encompass all cognitive domains), the systematic and frequent use of that rating scale in CRIS makes it an ideal proxy to compare with the NLP-CI score. For this subscale we derived a binary variable of a score of 1 or more (“HoNOS-CI-b”), in order to replicate best NLP-CI-b that potentially includes non-clinically significant impairments.

Participants and Clinical Data

Two samples were generated from CRIS for analysis:

1. Sample A (n = 24,614, 214,239 observations): patients with Sz active in CRIS between 2007 and 2020. This sample was used to analyze the prevalence of cognitive symptoms extracted by the NLP tool, along various demographic and clinical measures including age, gender, marital status, ethnicity, education, first language, employment, comorbidity with dementia, use of antipsychotic, antidepressant, and antidementia medications [following the method used in Bendayan et al. (36)]. We also used the following HoNOS subscales as covariates: “problems with activities of daily living” (HoNOS-ADL), “problems with relationships” (HoNOS-social), “problems associated with hallucinations and delusions” (HoNOS-positive symptoms), and “problems with depressed mood” (HoNOS-depression) (18). For all these HoNOS subscales, we derived binary variables based on a score of 2 or more, indicating clinically significant impairment (34).

2. Sample B (n = 12,234, 116,719 observations): the subset of patients from sample A who had at least 3 HoNOS scores recorded during their total length of care, following (37). This sample was used to model HoNOS trajectories and compare associations of cognition with demographic factors and clinical outcomes for both HoNOS and the NLP metric.

For each sample, the data was aggregated using 6 months buckets in order to derive the analyses (36).

Statistical Analysis

For both NLP-CI and HoNOS-CI scales, a threshold score of 0 was applied to measure the presence or absence of cognitive impairments (indicating that at least some level of cognitive dysfunction was recorded). This binary scale was used in conjunction with the ordinal scale to conduct analyses.

Logistic and linear regressions were used to assess the association of sociodemographic factors and clinical outcomes with the presence or absence of cognitive impairments. For clinical outcomes, the relation between cognition and number of hospital admissions and length of admission within 6 months of the score was estimated using linear regression. For socio-demographic covariates, missing data was treated as a separate category, and those showing a too high proportion of missing data or too low associations with both NLP-CI and HoNOS-CI were dropped from the analyses.

Finally, grouping was performed for certain variables when found to increase association or to reduce imbalance of sub-categories. Sensitivity analyses were performed to compare grouping methodologies and the grouping maximizing the association was selected.

For trajectories modeling, mixed linear models were used, with random slope and intercept. Linear and quadratic unconditional models were also examined, with and without random coefficients, but dropped in preference for the model maximizing the log-likelihood.

Results

Performance of NLP-CI

The performance of the NLP tool was evaluated using 10-fold cross-validation, using the manually annotated sentences described in section Measurement development. Further testing was conducted using a newly generated sample of 800 sentences, which were again manually annotated and compared against BioBert's classification labels independently.

The results from the cross validation analysis (averages on all folds for test and training sets, using weighted metrics) are summarized in Table 1A.

TABLE 1A

Table 1A. Performance of the NLP-CI tool for individual domains (average over 10-fold cross validation).

F1 scores ranged between 91 and 96% depending on the cognitive domains, and the overall application encompassing all symptoms showed a F1 score of 93%.

The independent blind testing, presented in Table 1B, showed similar performance results.

TABLE 1B

Table 1B. Performance of the NLP-CI tool for individual domains (separate blind set).

Prevalence of Cognitive Impairments and Cognitive Profiles

Table 2 shows the distribution of cognitive profiles by symptoms, as identified by NLP-CI. Of the 24,614 patients diagnosed with Sz in CRIS between 2007 and 2020, 55% had one or more mentions of cognitive problems in their clinical records, and 60% of patients with at least one problem recorded have mentions in all of the five domains identified. This supports the hypothesis that cognitive dysfunctions are a common feature of Sz.

TABLE 2

Table 2. Cognitive impairments profiles (Sample A).

The symptoms most frequently mentioned were executive function (present in 53% of patients from sample A), followed by memory (47%) and attention (44%).

Association With Socio-Demographic Factors

We ran binary logistic regression models to analyze the association of cognitive impairments with various socio-demographic and clinical factors (Table 3).

TABLE 3

Table 3. Analysis of factors associated with cognitive impairments in Sz (Sample B).

Several factors initially extracted (first language, antidementia medication, and antidepressant medication) included more than 30% of missing data and consequently were dropped from the analyses.

Furthermore, several categories were grouped to maximize association and/or reduce imbalance. For education, patients having a General Certificate of Secondary Education (GCSE), A-levels or University degrees were grouped together versus patients with no recorded education; for ethnicity, patients of non-white background were grouped in one category.

Results using NLP-CI indicate that patients with at least one symptom recorded are more likely to be male, of non-white background, living alone, and unemployed, which is broadly consistent with previous research in British aging cohorts (38). Additionally, the presence of cognitive symptoms was associated with problems with daily living, social impairment, and positive symptoms, with a potentially protective impact for antipsychotic medication. Patients with comorbid dementia were also more likely to show cognitive dysfunctions, supporting results from Bendayan et al. (36). The association of age with cognition appears inconclusive, and needs to be further disentangled by individual trajectories analysis.

HoNOS-CI shows a similar profile with differences for age and education, possibly due to coverage discrepancies (HoNOS is not used for children, and includes slightly different cognitive domains compared to the NLP application). Specifically, looking at a detailed breakdown of cognitive symptoms by education level (Table 4), attention and social cognition stand out as being more prevalent for patients with a qualification of GCSE or above compared to those without. This is further supported by the higher number of symptoms recorded for patients under the age of 40 in the unstructured portion of EHRs, and could be explained by the fact that certain cognitive dysfunctions, such as attention disorders, are more likely to be detected and dealt with in educational settings (39).

TABLE 4

Table 4. Prevalence of different NLP-CI symptoms grouped by education.

Association With Clinical Outcome (Number of Hospital Admissions and Length of Stay)

Table 5 and Figure 2 highlight the association of cognitive impairments with the number of hospital admissions and length of stay, within 6 months of mention of the problem. Regression analyses suggest that cognitive symptoms are associated with a higher probability of admission, as well as a longer length of hospitalization.

TABLE 5

Table 5. Association between number of cognitive impairments (estimated using NLP and HoNOS) with mental health hospital admission and duration of admission between 2007 and 2020 (Sample B).

FIGURE 2

Figure 2. Association of HoNOS and NLP scores with mental health stays.

Patients with at least one mention of cognitive problems in their medical record have on average 49% more hospital admissions within 6 months of the mention than those without, and the admission is on average 23 days longer.

These associations are mitigated by age, socio-demographic and clinical factors. Specifically, being female, employed, married or cohabitating, and not showing problems with daily living, relationships and positive symptoms reduces the frequency and length of hospitalizations. After full adjustment, the frequency of admissions is indeed reduced by more than half (patients with one or more cognitive problems have a 23% greater likelihood of admission) and the average length of stay by 9 days. Overall, results are broadly consistent between NLP-CI and HoNOS-CI.

Cognitive Symptoms Trajectories

Finally, mixed linear models were used to compare the cognitive trajectories of patients with Sz (using sample A for NLP-CI and sample B for HoNOS-CI, in order to maximize the number of datapoints and capture the complete evolution as recorded in EHRs), and their associations with various socio-demographic and clinical factors.

Here, we use antipsychotic treatment as a proxy for positive symptoms, given HoNOS scores are not recorded as frequently as cognitive symptoms are mentioned in unstructured text and consequently do not provide meaningful association when using sample A. In order to compare HoNOS-CI with NLP-CI, the baseline is defined as the date when the patient has their first mention recorded (on average 42 years old for HoNOS and 39 for NLP-CI).

Figure 3 illustrates the evolution of both scores over time, showing on average a constant slightly increasing slope.

FIGURE 3

Figure 3. Trajectories of HoNOS and NLP-based Cognitive Impairment scores (Sample B).

Results obtained with mixed linear models, summarized in Tables 6, 7, confirm most associations found using regression models for NLP-CI, specifically that patients who are male, of non-white background, educated, and living alone are more likely to show higher cognitive impairments when admitted and/or a faster worsening of the problems over time. After full adjustment, patients have on average 1.32 cognitive symptoms mentioned in EHRs at baseline, and this number increases by 0.05 every 6 months, indicating that cognitive problems worsen with age (resp. a severity of 0.96 at baseline and an increase of 0.06 when using HoNOS-CI).

TABLE 6

Table 6. Estimates for the fully adjusted linear mixed model for HoNOS-CI and NLP-CI trajectories.

TABLE 7

Table 7. Slope estimates for the fully adjusted linear mixed model for HoNOS-CI and NLP-CI trajectories.

The comparison between HoNOS-CI and NLP-CI models further informs the association of education and age with cognitive impairments. Both NLP and HoNOS scores indicate a slower decline in cognitive function for patients with education. However, educated patients have on average 5% more mentions of cognitive problems in the free text at baseline (which likely corresponds to when they are students). Again this may be explained by the greater coverage of cognitive domains and age groups with the NLP application, which could therefore potentially allow earlier detection of impairments, specifically for attention and emotion. Sensitivity analyses showed that removing “attention” and “emotion” from the cognitive domains covered (defining the new metric “NLP-CI adj”) and including patients only present in sample B attenuated the impact of education on the trajectories.

Discussion

Strengths and Limitations

Using an NLP method to extract symptoms from EHRs proves to be an effective and robust method, which does not require any specific training or wide-scale patient recruitment to take part in lengthy and costly tests. Our results show that cognitive impairments are frequently mentioned in medical records, and suggest associations generally in line with the HoNOS cognitive subscale. However, cognitive problems prove difficult to detect and assess, and are less routinely mentioned than positive symptoms, for which treatment remains a high priority in the British mental health system, and to a lesser extent negative symptoms (40). Furthermore, we assume that an absence of documentation represents an absence of symptoms, but another possibility could be that cognitive dysfunctions are secondary to positive symptoms and less frequently recorded. This has been partially mitigated in our study by the inclusion of positive symptoms as a covariate, but nonetheless may still result in underestimating the prevalence of cognitive problems when using the information extracted from free text compared to direct assessments.

Another limitation of our analysis was the use of a fixed set of keywords to identify symptoms, which could potentially exclude certain patients from detection, however, using a “free” approach would necessitate the annotation of a very large number of documents. Moreover, the keywords were derived from two major terminology sources (26, 27), which should cover most terms used by clinicians in medical records. This is further mitigated by the fact that the list was reviewed by clinicians and enriched by most common misspellings detected in CRIS electronic records [following the method proposed in Viani et al. (41)].

A key strength of our approach is the large sample covered, representative of patients with schizophrenia as well as the overall CRIS population. Using NLP allows us to rapidly extract information on large datasets, and the keywords approach proves generalizable for various types of symptoms and disorders (18, 23). This contrasts with most clinical assessments or rating scales, which are not often administered, or may not always capture all types of impairments. For instance HoNOS is routinely administered in the UK but is limited in its coverage of cognitive domains and population (34, 35). Despite these differences, the data extracted using the NLP framework shows a strong correlation with the HoNOS cognitive subscale in terms of symptoms trajectories, association with socio-demographic and clinical variables. We found cognitive dysfunction to be a common feature of patients with Sz, and to have a clear association with adverse clinical outcomes. Our data also highlights that cognitive impairments are an important driver for hospital admission, which is more traditionally linked to positive (42) or negative (18) symptoms.

Finally, the discrepancies related to age and education could indicate that clinicians perceive and document cognitive dysfunctions more readily when someone has previously completed a university degree but subsequently developed psychosis (assuming this occurred after university). This should be further explored in order to better assess the relationship between education and early detection of cognitive problems, in particular for attention and emotional impairments.

In summary, whilst NLP methods cannot provide results as accurate as clinical assessments or rating scales, they present the advantage of being cost-effective and cover a larger population as well as a broader and more granular timespan. Consequently, NLP results can be seen as complementary and used to screen patients showing cognitive problems, who could in turn benefit from more specific tests (e.g., by being selected to take part in neurocognitive batteries studies) as well as targeted treatment.

Future Research and Conclusions

Our findings suggest that cognitive impairments can reliably be extracted from clinical records using NLP methods, and reveal their high prevalence in patients with Sz as well as a clear association with poor clinical outcomes.

Analyzing data from the free text of EHRs provides broad coverage in terms of the number of patients and age range, in contrast to neurocognitive batteries or rating scales. Therefore, this can facilitate systematic patient screening and large scale and early detection of cognitive problems. Furthermore, NLP tools allow us to extract a number of socio-demographic and clinical parameters not recorded in the structured portion of EHRs which, combined together, can provide new insights into the onset and development of symptoms and eventually inform clinical decisions.

This reveals the potential of such automated tools to harvest meaningful information from the unstructured text of medical records, and in this specific study highlights the importance of developing more targeted and effective treatments for patients with Sz suffering from cognitive impairments.

Data Availability Statement

The datasets presented in this article are not readily available because they contain participants identifiable data. Requests to access the datasets should be directed to YXVyZWxpZS5tYXNjaW8mI3gwMDA0MDtrY2wuYWMudWs=.

Author Contributions

AM conceived the study, conducted the data analysis, and drafted the manuscript. RS, AR, RP, and TP participated in the study design and interpretation of the results. RB, MW, and LM helped design the annotation guidelines and annotated the clinical data. All authors commented on drafts of the manuscript and approved the final version.

Funding

AM is funded by Takeda California, Inc. This work was supported by Health Data Research UK, an initiative funded by UK Research and Innovation, Department of Health and Social Care (England) and the devolved administrations, and leading medical research charities. This paper represents independent research part funded by the National Institute for Health Research (NIHR) Biomedical Research Center at South London and Maudsley NHS Foundation Trust and King's College London. The funders were not involved in the study design, collection, analysis, interpretation of data, the writing of this article or the decision to submit it for publication.

Disclaimer

The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank the reviewers for their valuable suggestions. This work uses data provided by patients and collected by the NHS as part of their care and support. The Clinical Record Interactive Search (CRIS) application was approved as a database for secondary analysis by the Oxford Research Ethics Committee (18/SC/0372). The work undertaken as part of this project was approved by the CRIS Oversight Committee.

References

1. Charlson FJ, Ferrari AJ, Santomauro DF, Diminic S, Stockings E, Scott JG, et al. Global epidemiology and burden of schizophrenia: findings from the Global Burden of Disease Study 2016. Schizophr Bull. (2018) 44:1195–203. doi: 10.1093/schbul/sby058

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Hor K, Taylor M. Suicide and schizophrenia: a systematic review of rates and risk factors. J Psychopharmacol Oxf Engl. (2010) 24:81–90. doi: 10.1177/1359786810385490

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Laursen TM, Munk-Olsen T, Vestergaard M. Life expectancy and cardiovascular mortality in persons with schizophrenia. Curr Opin Psychiatry. (2012) 25:83–8. doi: 10.1097/YCO.0b013e32835035ca

PubMed Abstract | CrossRef Full Text | Google Scholar

4. World Health Organization. WHO | The World Health Report 2001 – Mental Health: New Understanding. New Hope, MN: WHO (2001). Available online at: https://www.who.int/whr/2001/en/ (cited May 6, 2021).

Google Scholar

5. Bora E, Yucel M, Pantelis C. Cognitive functioning in schizophrenia, schizoaffective disorder and affective psychoses: meta-analytic study. Br J Psychiatry J Ment Sci. (2009) 195:475–82. doi: 10.1192/bjp.bp.108.055731

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Heinrichs RW, Zakzanis KK. Neurocognitive deficit in schizophrenia: a quantitative review of the evidence. Neuropsychology. (1998) 12:426–45. doi: 10.1037/0894-4105.12.3.426

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Bleuler E. Dementia Praecox or the Group of Schizophrenias. Oxford: International Universities Press (1950). 548 p.

Google Scholar

8. Tripathi A, Kar SK, Shukla R. Cognitive deficits in schizophrenia: understanding the biological correlates and remediation strategies. Clin Psychopharmacol Neurosci. (2018) 16:7–17. doi: 10.9758/cpn.2018.16.1.7

PubMed Abstract | CrossRef Full Text | Google Scholar

9. August SM, Kiwanuka JN, McMahon RP, Gold JM. The MATRICS Consensus Cognitive Battery (MCCB): clinical and cognitive correlates. Schizophr Res. (2012) 134:76–82. doi: 10.1016/j.schres.2011.10.015

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Sahakian BJ, Owen AM. Computerized assessment in neuropsychiatry using CANTAB: discussion paper. J R Soc Med. (1992) 85:399–402.

PubMed Abstract | Google Scholar

11. Smith PJ, Need AC, Cirulli ET, Chiba-Falek O, Attix DK. A comparison of the Cambridge Automated Neuropsychological Test Battery (CANTAB) with “traditional” neuropsychological testing instruments. J Clin Exp Neuropsychol. (2013) 35:319–28. doi: 10.1080/13803395.2013.771618

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Levaux M-N, Potvin S, Sepehry AA, Sablier J, Mendrek A, Stip E. Computerized assessment of cognition in schizophrenia: promises and pitfalls of CANTAB. Eur Psychiatry. (2007) 22:104–15. doi: 10.1016/j.eurpsy.2006.11.004

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Keefe. The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophr Res. (2004) 68:283–97. doi: 10.1016/j.schres.2003.09.011

CrossRef Full Text | Google Scholar

14. Mehta N, Pandit A. Concurrence of big data analytics and healthcare: a systematic review. Int J Med Inf. (2018) 114:57–65. doi: 10.1016/j.ijmedinf.2018.03.013

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Koleck TA, Dreisbach C, Bourne PE, Bakken S. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. J Am Med Inform Assoc. (2019) 26:364–79. doi: 10.1093/jamia/ocy173

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Iqbal E, Mallah R, Jackson RG, Ball M, Ibrahim ZM, Broadbent M, et al. Identification of adverse drug events from free text electronic patient records and information in a large mental health case register. PLoS One. (2015) 10:e0134208. doi: 10.1371/journal.pone.0134208

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Patel R, Lloyd T, Jackson R, Ball M, Shetty H, Broadbent M, et al. Mood instability is a common feature of mental health disorders and is associated with poor clinical outcomes. BMJ Open. (2015) 5:e007504. doi: 10.1136/bmjopen-2014-007504

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Patel, Jayatilleke N, Broadbent M, Chang C-K, Foskett N, Gorrell G, et al. Negative symptoms in schizophrenia: a study in a large clinical sample of patients using a novel automated method. BMJ Open. (2015) 5:e007619. doi: 10.1136/bmjopen-2015-007619

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ramu N, Kolliakou A, Sanyal J, Patel R, Stewart R. Recorded poor insight as a predictor of service use outcomes: cohort study of patients with first-episode psychosis in a large mental healthcare database. BMJ Open. (2019) 9:e028929. doi: 10.1136/bmjopen-2019-028929

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Birnie KI, Stewart R, Kolliakou A. Recorded atypical hallucinations in psychotic and affective disorders and associations with non-benzodiazepine hypnotic use: the South London and Maudsley Case Register. BMJ Open. (2018) 8:e025216. doi: 10.1136/bmjopen-2018-025216

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Liu Q, Woo M, Zou X, Champaneria A, Lau C, Mubbashar MI, et al. Symptom-based patient stratification in mental illness using clinical notes. J Biomed Inf. (2019) 98:103274. doi: 10.1016/j.jbi.2019.103274

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Stewart R, Soremekun M, Perera G, Broadbent M, Callard F, Denis M, et al. The South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLAM BRC) case register: development and descriptive data. BMC Psychiatry. (2009) 9:51. doi: 10.1186/1471-244X-9-51

PubMed Abstract | CrossRef Full Text

23. Perera G, Broadbent M, Callard F, Chang C-K, Downs J, Dutta R, et al. Cohort profile of the South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLaM BRC) Case Register: current status and recent enhancement of an Electronic Mental Health Record-derived data resource. BMJ Open. (2016) 6:e008721. doi: 10.1136/bmjopen-2015-008721

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Keefe RSE, Poe M, Walker TM, Kang JW, Harvey PD. The schizophrenia cognition rating scale: an interview-based assessment and its relationship to cognition, real-world functioning, and functional capacity. Am J Psychiatry. (2006) 163:426–32. doi: 10.1176/appi.ajp.163.3.426

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Stip E, Caron J, Renaud S, Pampoulova T, Lecomte Y. Exploring cognitive complaints in schizophrenia: the subjective scale to investigate cognition in schizophrenia. Compr Psychiatry. (2003) 44:331–40. doi: 10.1016/S0010-440X(03)00086-5

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Stearns MQ, Price C, Spackman KA, Wang AY. SNOMED clinical terms: overview of the development process and project status. in Proceedings of the AMIA Symposium. American Medical Informatics Association (2001). p. 662.

PubMed Abstract | Google Scholar

27. Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. (2004) 32:D267–70. doi: 10.1093/nar/gkh061

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Mascio A, Kraljevic Z, Bean D, Dobson R, Stewart R, Bendayan R, et al. Comparative analysis of text classification approaches in electronic health records. arXiv200506624 [Preprint]. (2020). doi: 10.18653/v1/2020.bionlp-1.9

CrossRef Full Text | Google Scholar

29. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv181004805 [Preprint]. (2019). doi: 10.18653/v1/N19-1423

CrossRef Full Text | Google Scholar

30. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. HuggingFace's transformers: state-of-the-art natural language processing. arXiv191003771 [Preprint]. (2020).doi: 10.18653/v1/2020.emnlp-demos.6

CrossRef Full Text | Google Scholar

31. Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. (2019) 36:1234–40. doi: 10.1093/bioinformatics/btz682

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Sun C, Qiu X, Xu Y, Huang X. How to fine-tune BERT for text classification? In: Sun M, Huang X, Ji H, Liu Z, Liu Y, editors. Chinese Computational Linguistics, Vol. 11856, Lecture Notes in Computer Science. Cham: Springer International Publishing (2019). p. 194–206.

Google Scholar

33. Powers DMW. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv201016061 [Preprint]. (2020).

Google Scholar

34. Wing JK, Beevor AS, Curtis RH, Park SGB, Hadden J, Burns A. Health of the Nation Outcome Scales (HoNOS): research and development. Br J Psychiatry. (1998) 172:11–8. doi: 10.1192/bjp.172.1.11

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Crawford MJ, Zoha M, Macdonald AJD, Kingdon D. Improving the quality of mental health services using patient outcome data: making the most of HoNOS. BJPsych Bull. (2017) 41:172–6. doi: 10.1192/pb.bp.116.054346

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Bendayan R, Mascio A, Stewart R, Roberts A, Dobson RJ. Cognitive trajectories in comorbid dementia with schizophrenia or bipolar disorder: the South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLaM BRC) Case Register. Am J Geriatr Psychiatry. (2021) 29:604–16. doi: 10.1016/j.jagp.2020.10.018

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Singer JD, Willett JB, Willett CWEPJB, Willett JB. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford: Oxford University Press (2003). 672 p.

Google Scholar

38. Bendayan R, Piccinin AM, Hofer SM, Cadar D, Johansson B, Muniz-Terrera G. Decline in memory, visuospatial ability, and crystalized cognitive abilities in older adults: normative aging or terminal decline? J Aging Res. (2017) 2017:6210105. doi: 10.1155/2017/6210105

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Harper G. The ADHD Explosion: myths, medication, money, and today's push for performance. Psychiatr Serv. (2016) 67:e5. doi: 10.1176/appi.ps.670604

CrossRef Full Text | Google Scholar

40. Jacobs R, Barrenho E. Impact of crisis resolution and home treatment teams on psychiatric admissions in England. Br J Psychiatry J Ment Sci. (2011) 199:71–6. doi: 10.1192/bjp.bp.110.079830

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Viani N, Patel R, Stewart R, Velupillai S. Generating positive psychosis symptom keywords from electronic health records. In: Riaño D, Wilk S, ten Teije A, editors. Artificial Intelligence in Medicine, Vol. 11526, Lecture Notes in Computer Science. Cham: Springer International Publishing (2019). p. 298–303.

Google Scholar

42. Olfson M, Ascher-Svanum H, Faries DE, Marcus SC. Predicting psychiatric hospital admission among adults with schizophrenia. Psychiatr Serv. (2011) 62:1138–45. doi: 10.1176/ps.62.10.pss6210_1138

PubMed Abstract | CrossRef Full Text | Google Scholar