Gabriele Cattaneo1,2,3*
Alba Roca-Ventura1,3,4
Eva Heras5Maria Anglada5Jan Missé5Encarnació Ulloa5
Simon Fankhauser1,2,3
Eloy Opisso1,2,3Alberto García-Molina1,2,3
Javier Solana-Sánchez1,2,3- 1Institut Guttmann, Institut Universitari de Neurorehabilitació Adscrit a la UAB, Badalona, Spain
- 2Departament de Medicina, Universitat Autònoma de Barcelona, Bellaterra, Spain
- 3Fundació Institut d'Investigació en Ciències de la Salut Germans Trias i Pujol, Barcelona, Spain
- 4Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
- 5Servei Envelliment i Salut Servei Andorrà d'Atenció Sanitària, Andorra la Vella, Andorra
Introduction: Digital solutions for cognitive assessment are currently not only widely used in experimental contexts but can also be useful in clinical practice for efficient screening and longitudinal follow-up. The “Guttmann Cognitest”®, which includes seven computerized tasks designed to assess main cognitive functions, revealed in a previous validation study to be a potential useful tool to assess cognitive functioning in healthy middle-aged adults.
Method: Here, we present results from a validation in two different populations: one consisting of older adults, and the other comprising young and middle-aged individuals, some of them affected by acquired brain injury. To perform a convergent validity test, older adults were also administered with the MOCA, while young and middle-aged individuals were administered with a short neuropsychological assessment including gold-standard neuropsychological tests. We also conducted sensitivity and specificity analysis to establish the utility of this instrument in identifying potential cognitive dysfunctions in the two groups.
Results: Results demonstrated strong convergent validity as well as good specificity and sensitivity characteristics.
Discussion: This tool is a valid and useful instrument to assess cognitive functioning and detecting potential cases of cognitive dysfunctions in older adults and clinical populations.
1 Introduction
As life expectancy continues to rise, healthcare services around the world will be progressively and increasingly challenged by age-related diseases (1). It has been estimated that the proportion of people over 55 years will surpass people under 15 years by the year of 2035 (2). Advancing age represents the main risk factor for the development of brain related diseases (3).
For example, Alzheimer's disease and other forms of dementias is the 7th causes of death worldwide, the second in high income countries (4, 5). Estimations on disability further propose that in 2050, half of the global burden due to disability will be attributable to mental and cognitive disorders (6). Additionally, aging itself is recognized to lead to the decline or alteration of structural and functional brain mechanisms (7), resulting in subsequent functional impact.
In this context early detection of cognitive decline is crucial to implement preventive strategies that can halt the progression of decline and promote brain health maintenance (8, 9). To achieve this aim there is the need to implement efficient, and accessible cognitive assessments that can allow us to identify preclinical stages of diseases and detect subtle cognitive changes over time. This is especially relevant for Alzheimer's disease and other neurodegenerative pathologies preceded by a long preclinical phase. The early detection of these cases could have significant implications for individuals' quality of life and level of independence afterward. Once detected and diagnosed, these conditions could be accompanied by an appropriate and efficient follow-up for monitor cognitive changes and evaluate the potential effects of prescribed interventions.
Unfortunately, the current reality is that Alzheimer's disease is largely underdiagnosed (10). This underdiagnosis is even more pronounced for preclinical subtle symptoms due to limitations in primary care, such as a lack of specially qualified personnel, resources, and optimal tools. Recent data showed that even if 96% of physicians express the desire to screen for cognitive functioning, 50% of older adults undergo effective examinations, and only 15% periodically (11).
Addressing these challenges and improving the rates of detection and follow-up is essential to ensure timely intervention and support for individuals at risk of cognitive decline. By enhancing access to appropriate assessment tools, it will become possible to increase the identification of cognitive changes at an early stage and enable the implementation of preventive measures. It has been estimated that delaying the onset of symptoms of dementia of only 1 year could prevent in over 11.8 million cases in the next 30 years, reducing expenses for healthcare of $219 billion (12, 13).
In this scenario, mobile technologies and computerized cognitive assessment may represent the solution (14).
This kind of tools have been developed and studied for years, showing to be a usable and reliable solution. They offer the potential to perform population based longitudinal screening that will allow to early detect cognitive changes, without incurring the costs associated with administering in-person neuropsychological tests (11).
Following the same logic, these tools could be extremely useful also in clinical contexts to monitor people diagnosed with brain pathologies, as acquired brain injuries (ABI) (15). It could allow to optimize human and economic resources and improve healthcare quality by simplifying and speeding up patient's assessments, and maximizing patient's monitoring possibilities, as done for non-cognitive outcomes [see for example (16)].
However, in the specific contexts of people with ABI the use of this digital solution could be complicated by patient's characteristics and heterogeneity. Concretely, the usability of these solutions could be potentially reduced due to motor, visual, comprehension, or other pathology's related impairments that difficult or totally prevent tasks execution. In these cases, the administration would probably need to be filtered and guided by a health care professional.
In a previous study, we explored the validity and usability of the “Guttman Cognitest,” a digital solution to assess cognitive functioning, in middle aged and cognitively unimpaired adults (17) and calculated regression-based norms.
Here, we aim to extend previous findings further exploring convergent validity. Moreover, we studied the capacity of this tool to detect possible cases of cognitive impairment.
2 Materials and methods
2.1 Participants
A total of 318 young and middle-aged adults (141 women; mean age = 54.5, range = 19–69) and 71 older adults over 70 years (43 women; mean age = 76.7, range = 70–88) participated in this study (see Table 1).
Table 1. Sociodemographic characteristics of participants.
The group of older adults were composed of people recruited from Andorra and participating in the Integrated care for older people (ICOPE) program promoted by the WHO. This program is part of a population-based study conducted by the Andorran Health Care System with the aim of identifying population frailty. In this group 39 subjects presented MCI according with classically used clinical criteria [MOCA < 26; (18–20)], while 32 participants scored 26 or more.
Young and middle-aged adults instead were patients of the Neurorehabilitation Hospital Institut Guttmann, and people participating in the Barcelona Brain Health Initiative (8).
In this group 44 participant were affected by ABI (See Table 2 for etiology of brain injuries), and between them 42 resulted cognitively impaired in at least one cognitive function.
Table 2. Etiology of acquired brain injury in patients of Neurorehabilitation Hospital Institut Guttmann.
All participants provided explicit informed consent, and the protocol was approved by the Ethics and Clinical Research Committee of the Catalan Hospitals Union (Comité d'Ètica I Investigació Clínica de la Unió Catalana hospitals).
Participants with motor and visual impairments, as well as comprehension deficits that could alter tasks execution were excluded from this study.
2.2 Procedures
Older adults were administered with the “Guttmann Cognitest”® and the Montreal Cognitive Assessment (MOCA) (21) on the same day. On the other hand, young and middle-aged adults were administered with the “Guttmann Cognitest”® and a neuropsychological test battery, by expert's neuropsychologist, on different days (mean difference between tests administration in days = 4.1; see Table 3 for results). The administration of the digital solution in people with ABI and older adults was supervised by an expert neuropsychologist to assure that they did not present visual, physical and comprehension impairments that could alter their results.
Table 3. Results of the neuropsychological tests for older adults and people with ABI.
The neuropsychological assessment was designed to measure visuo-spatial searching, and attention [Trail making test A; (22)], working memory [Digit Span forward; (23)] and episodic memory [Rey Auditory Verbal Learning Test (RAVLT) (24)].
The “Guttmann Cognitest”® includes 7 tasks designed to assess memory, executive functions, and visuo-spatial abilities [see (17) for a detailed description of the tasks].
All tasks are preceded by a set of screens with detailed instructions together with a video tutorial, explaining the objective of the task and the expected behavior of the user.
After this, a simple demo screen of the task plays as practice for the user, to ensure that they have understood the logic, objectives and expected responses of the task. Only once the practice is completed correctly (there are two attempts) the corresponding task begins. If the practice is not completed correctly the task is not administered and the system moves directly to the next task, assuming that the person would not have been able to complete that task correctly due to task comprehension.
2.3 Data and statistical analysis
Following the same procedure as reported in previous studies [e.g., (17, 25–27)], we transformed raw scores obtained in gold-standard neuropsychological tests into z-scores and then calculated a global-cognition score as their mean.
Z-scores for the “Guttmann Cognitest”® were calculated using the formulae estimates with the regression-based norms previously published (17), and global cognition composite score was determined by taking the mean of all the transformed z-scores. As typical norms for neuropsychological classical tests these formulae correct the raw score for age, biological sex and educational level to control for their effects on the results.
Convergent validity was assessed using Spearman's rank correlations between the global composite score of the “Guttmann Cognitest”® and results obtained in the classical neuropsychological tests (MOCA for older adults, and global composite score for young and middle-aged adults).
We then performed a receiver operating characteristic curve in both groups (ROC curve) to estimate the value that showed the best sensitivity and specificity characteristics to detect cognitive impairments. This was defined as a score inferior to 26 at the MOCA (18–20), or people with ABI that scored below the cut-off usually employed for clinical practice, and based on Spanish normative data, in at least one of the classical neuropsychological tests (42 out of 44 patients).
All statistical analyses were performed in R version 4.2.1 (28), and run in RStudio, version 2023.06.1 (29).
3 Results
3.1 Convergent validity
3.1.1 Older adults
The Global “Cognitest” composite score showed strong correlation with the MOCA score (rs = 0.71; p < 0.001) indicating good convergent validity (Figure 1).
Figure 1. Scatterplot representing the correlation between MOCA scores and global cognition composite scores obtained with the Cognitest.
3.1.2 Young and middle-aged adults
Also, in this group we found a strong correlation between Global Cognitest composite score, and Global composite score calculated for classical paper and pencil neuropsychological tests (rs = 0.60; p < 0.001), confirming good convergent validity (Figure 2).
Figure 2. Scatterplot representing the correlation between global cognition scores obtained with the standard tests, and global cognition composite scores obtained with the Cognitest.
3.2 Receiver operating characteristic curve
3.2.1 Older adults
The analysis revealed an area under the curve of 0.84 (CI 95%: 0.75–0.93), indicating a satisfactory discriminant capacity. According to Youden's J statistics, the optimal discriminant Global Cognitest z-score value was found to be−0.70, resulting in a sensitivity of 0.72 and a specificity of 0.91 (Youden's index = 0.62; see Figure 3).
Figure 3. ROC curve for detecting MCI in older adults. The light blue shape represents the CI 95%.
This cut-off value correctly identified 28 out of 39 individuals presenting MCI and only resulted in 3 false positive out of 32 cases.
3.2.2 Young and middle-aged adults
The area under the curve was 0.98 (CI 95%: 0.96–0.99), indicating a very good discriminant capacity. With a z-score of −0.85, sensitivity was 0.93 and a specificity 0.95 (Youden's index = 0.87; see Figure 4).
Figure 4. ROC curve for detecting cognitive impairments due to ABI in young and middle-aged adults. The light blue shape represents the CI 95%.
This cut-off value correctly identified 39 out of 42 individuals presenting cognitive impairments and produced only 15 false positive out of 276 unimpaired individuals.
4 Discussion
In this study, we aimed to explore convergent validity between global cognitive performance measured by classical neuropsychological tests, and by the Guttmann Cognitest digital solution in older adults and a group of young and middle adults. Moreover, we studied its utility in detecting in these groups, respectively possible cases of MCI and cognitive dysfunctions in people presenting ABI.
Results showed strong correlation between a global score obtained on the Cognitest and global cognitive functioning calculated with classical paper and pencil neuropsychological tests. This indicates a very good convergent validity in both older adults and young and middle-aged adults.
Present findings are in line with previous validation study of computerized neuropsychological assessment tools in healthy adults and clinical populations [see (11, 15, 30)], and our previous validation (17). These studies demonstrated some variability in terms of convergent and concurrent validity, ranging from small to large effect sizes (from 0.2 to 0.88).
The variability observed in these results makes it somewhat challenging to establish clear expectations regarding the desired effect sizes for an effective solution.
However, considering the results obtained, the Guttmann Cognitest exhibited overall a good convergent validity when compared to established neuropsychological tests considered as the gold standard.
Crucially, the receiver operating characteristic analysis showed a specificity and sensitivity characteristics in line with those of other classical screening tests used to detect cognitive impairments and MCI [see (31–33) for review].
This result demonstrate that this digital solution represent a potentially useful tool to be employed in clinical populations to identify mild cognitive impairment (34) and other cognitive dysfunctions.
The present study presented some limitations that future validation studies must consider and solve. We did not screen participants for depression, that could have biased their performance at the tests. Moreover, two of the three memory tasks (Cued image-number association and long term memory) showed significant ceiling effects, showing to be too easy, and with potential reduced discriminant capacity (See Supplementary material for details). However, if we consider the whole tasks, as for calculating “Global cognition” composite scores, the ceiling effect is reduced.
Taken together, these results indicate that digital solutions, such as ours, have the potential to serve as a benchmark for conducting large-scale population screening, and a useful tool for clinics to increase healthcare quality. Indeed, this can effectively reduce the underdiagnosis of cognitive impairment and potentially detect preclinical cognitive changes in aging, as well as help for patient's cognitive screening.
Together with its potential to efficiently monitor cognition over time, these innovative procedures will allow to improve patient's follow-up and implement preventive strategies. This can reduce the progression of cognitive decline and promote resilience, having an important impact on people's health and wellbeing, and an enormous economic impact on society.
In conclusion, our findings demonstrate the utility of digital solutions like the “Guttmann Cognitest”® as a reliable and efficient tool for evaluating cognitive functions in older adults and clinical populations.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving humans were approved by the Comite d'Etica de la Unió Catalana d'Hospitals. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
GC: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing—original draft. AR-V: Conceptualization, Investigation, Writing—review & editing. EH: Conceptualization, Investigation, Supervision, Writing—review & editing. MA: Data curation, Investigation, Writing—review & editing. JM: Investigation, Writing—review & editing. EU: Investigation, Writing—review & editing. SF: Investigation, Writing—review & editing. EO: Conceptualization, Investigation, Supervision, Writing—review & editing. AG-M: Conceptualization, Investigation, Supervision, Writing—review & editing. JS-S: Conceptualization, Investigation, Supervision, Writing—review & editing.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. GC is partially funded by the Fundació La Marató De TV3 (202211_30) and the Alzheimer Drug Discovery Foundation (RDADB-202212–2024379). JS-S is partially funded by the Fundació La Marató De TV3 (202211_30) and the Alzheimer Drug Discovery Foundation (RDADB-202212–2024379).
Acknowledgments
The authors extend their gratitude to all participants, for their invaluable collaboration. No companies were involved in any step of the study, from design to data collection and analysis, data interpretation, the writing of the article or the decision to submit it for publication.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur.2023.1292960/full#supplementary-material
References
1. Santosa A. A better world towards convergence of longevity? Lancet. (2017) 389:1278–9. doi: 10.1016/S0140-6736(17)30314-8
2. Harasty C, Ostermeier M. Population Ageing: Alternative Measures of Dependency and Implications for the Future of Work. Geneva: International Labour Organization (2020).
3. Barnett K, Mercer SW, Norbury M, Watt G, Wyke S, Guthrie B. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet. (2012) 380:37–43. doi: 10.1016/S0140-6736(12)60240-2
4. World Health Organization. Dementia (2023). Available online at: https://www.who.int/news-room/fact-sheets/detail/dementia
5. Alzheimer's Association. 2023 Alzheimer's disease facts and figures. Alzheimers Dement. (2023) 19:1598–695. doi: 10.1002/alz.13016
6. Mathers CD, Loncar D. Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med. (2006) 3:2011–30. doi: 10.1371/journal.pmed.0030442
7. Nyberg L, Lövdén M, Riklund K, Lindenberger U, Bäckman L. Memory aging and brain maintenance. Trends Cogn Sci. (2012) 16:292–305. doi: 10.1016/j.tics.2012.04.005
8. Cattaneo G, Bartrés-faz D, Morris TP, Sánchez JS, Macià D, Tarrero C, Pascual-Leone A. The Barcelona brain health initiative : a cohort study to define and promote determinants of brain health. Front. Aging Neurosci. (2018) 10:321. doi: 10.3389/fnagi.2018.00321
9. Nyberg L, Pudas S. Successful memory aging. Annu Rev Psychol. (2019) 70:219–43. doi: 10.1146/annurev-psych-010418-103052
10. Lang L, Clifford A, Wei L, Zhang D, Leung D, Augustine G, et al. Prevalence and determinants of undetected dementia in the community: a systematic literature review and a meta-analysis. BMJ Open. (2017) 7:1–8. doi: 10.1136/bmjopen-2016-011146
11. Tsoy E, Zygouris S, Possin KL. Current state of self-administered brief computerized cognitive assessments for detection of cognitive disorders in older adults: a systematic review. J Prev. Alzheimer's Dis. (2021) 8:267–276. doi: 10.14283/jpad.2021.11
12. Brookmeyer R, Johnson E, Ziegler-Graham K, Arrighi HM. Forecasting the global burden of Alzheimer's disease. Alzheimer's Dementia. (2007) 3:186–91. doi: 10.1016/j.jalz.2007.04.381
13. Zissimopoulos J, Crimmins E, Stclair P. The value of delaying alzheimer's disease onset. Forum Health Econ Policy. (2015) 18:25–39. doi: 10.1515/fhep-2014-0013
14. Cattaneo G, Solana-Sanchez J, Tormos-Muñoz JM. Remote cognitive assessment to early detection of cognitive decline. Aging. (2023) 15:5961–2. doi: 10.18632/aging.204908
15. Sloane KL, Fabian R, Wright A, Saxena S, Kim K, Stein CM, et al. Supervised, self-administered tablet-based cognitive assessment in neurodegenerative disorders and stroke. Dement Geriatr Cogn Disord. (2023) 52:74–82. doi: 10.1159/000527060
16. Smith BG, Tumpa S, Mantle O, Whiffin CJ, Mee H, Solla DJF, et al. Remote follow-up technologies in traumatic brain injury: a scoping review. J Neurotrauma. (2022) 39:1289–317. doi: 10.1089/neu.2022.0138
17. Cattaneo G, Pachón-García C, Roca A, Alviarez-Schulze V, Opisso E, García-Molina A, et al. “Guttmann Cognitest”®, preliminary validation of a digital solution to test cognitive performance. Front Aging Neurosci. (2022) 14:1267. doi: 10.3389/fnagi.2022.987891
18. Aguilar-Navarro SG, Mimenza-Alvarado AJ, Palacios-García AA, Samudio-Cruz A, Gutiérrez-Gutiérre LA, Ávila-Funes JA. Validity and reliability of the Spanish version of the Montreal Cognitive Assessment (MoCA) for the detection of cognitive impairment in Mexico. Rev Colomb Psiquiatr. (2018) 47:237–43. doi: 10.1016/j.rcp.2017.05.003
19. Milani SA, Marsiske M, Cottler LB, Chen X, Striley CW. Optimal cutoffs for the montreal cognitive assessment vary by race and ethnicity. Alzheimer's Dementia Diag Assess Dis Monitor. (2018) 10:773–81. doi: 10.1016/j.dadm.2018.09.003
20. Pereiro AX, Ramos-Lema S, Lojo-Seoane C, Guàrdia-Olmos J, Facal-Mayo D, Juncos-Rabadán O, et al. Normative data for the montreal cognitive assessment (MOCA) in a Spanish sample of community-dweller adults. Eur Geriatr Med. (2017) 8:240–4. doi: 10.1016/j.eurger.2017.04.003
21. Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The montreal cognitive assessment, moca: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. (2005) 53:695–9. doi: 10.1111/j.1532-5415.2005.53221.x
22. Reitan RM, Wolfson D. The Halstead-Reitan Neuropsychological Test Battery: Theory and Interpretattion. Tucson, AZ: Neuropsychology Press (1985).
24. Rey A. L'examen psychologique dans les cas d'encéphalopathie traumatique. (Les problems) [The psychological examination in cases of traumatic encepholopathy Problems]. Archives Psychol. (1941) 28:215–85.
25. España-Irla G, Gomes-Osman J, Cattaneo G, Albu S, Cabello-Toscano M, Solana-Sanchéz J, et al. Associations between cardiorespiratory fitness, cardiovascular risk, and cognition are mediated by structural brain health in midlife. J Am Heart Assoc. (2021) 10:34514813. doi: 10.1161/JAHA.120.020688
26. Hinchman CA, Cabral DF, Ciesla M, Flothmann M, Nunez C, Rice J, et al. Exercise engagement drives changes in cognition and cardiorespiratory fitness after 8 weeks of aerobic training in sedentary aging adults at risk of cognitive decline. Front Rehabil Sci. (2022) 3. doi: 10.3389/fresc.2022.923141
27. Redondo-Camós M, Cattaneo G, Perellón-Alfonso R, Alviarez-Schulze V, Morris TP, Solana-Sanchez J, et al. Local prefrontal cortex TMS-induced reactivity is related to working memory and reasoning in middle-aged adults. Front Psychol. (2022) 13:290. doi: 10.3389/fpsyg.2022.813444
28. R Core Team. R: A Language Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing (2022). Available online at: https://www.R-project.org
30. Gualtieri CT, Johnson LG. Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Arch Clin Neuropsychol. (2006) 21:623–43. doi: 10.1016/j.acn.2006.05.007
31. Bello-Lepe S, Alonso-Sánchez MF, Ortega A, Gaete M, Veliz M, Lira J, et al. Montreal cognitive assessment as screening measure for mild and major neurocognitive disorder in a chilean population. Dement Geriatr Cogn Dis Extra. (2020) 10:105–14. doi: 10.1159/000506280
32. Delgado C, Araneda A, Behrens MI. Validación del instrumento montreal cognitive assessment en español en adultos mayores de 60 años. Neurología. (2019) 34:376–85. doi: 10.1016/j.nrl.2017.01.013
33. Tsoi KKF, Chan JYC, Hirai HW, Wong A, Mok VCT, Lam LCW, et al. Recall tests are effective to detect mild cognitive impairment: a systematic review and meta-analysis of 108 diagnostic studies. J Am Med Dir Assoc. (2017) 18:e17–807. doi: 10.1016/j.jamda.2017.05.016
Keywords: aging, cognitive decline, cognitive functioning, computerized cognitive assessment, memory
Citation: Cattaneo G, Roca-Ventura A, Heras E, Anglada M, Missé J, Ulloa E, Fankhauser S, Opisso E, García-Molina A and Solana-Sánchez J (2024) Investigating the application of “Guttmann Cognitest”® in older adults and people with acquired brain injury. Front. Neurol. 14:1292960. doi: 10.3389/fneur.2023.1292960
Received: 14 September 2023; Accepted: 13 December 2023;
Published: 08 January 2024.
Edited by:
Manish Kumar, University of Alabama at Birmingham, United StatesReviewed by:
Lydia Kaoutzani, Augusta University, United StatesAnkit Seth, Augusta University, United States
Copyright © 2024 Cattaneo, Roca-Ventura, Heras, Anglada, Missé, Ulloa, Fankhauser, Opisso, García-Molina and Solana-Sánchez. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Gabriele Cattaneo, gcattaneo@guttmann.com