Qi Zhang1†
Jia-Hang Wei2†
Xue Fu3†
Xin Liu4,5Xin-Yi Li1Wei Liu1
Zhong-Liang Liu1
Xiao-Qin Duan1,5*‡
Bin Zheng5*‡- 1Department of Rehabilitation Medicine, Jilin University Second Hospital, Changchun, China
- 2Jilin University, Changchun, China
- 3Changchun University of Chinese Medicine, Changchun, China
- 4School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China
- 5Surgical Simulation Research Lab, Department of Surgery, University of Alberta, Edmonton, AB, Canada
Purpose: To summarize the classification of computerized cognitive assessment (CCA) tools for assessing stroke patients, to clarify their benefits and limitations, and to reveal strategies for future studies on CCA tools.
Methods: A literature review was performed using PubMed, Embase, Scopus, JAMA Network, Cochrane Library and PsycINFO databases from January 1st, 2010, to August 1st, 2022. Two authors independently screened the literature following the same criteria, evaluated the study quality, and collected data from the articles.
Results: A total of 8,697 papers were acquired from the six databases. A total of 74 potentially eligible articles were selected for review. Of these, 29 articles were not relevant to this research, 3 were reviews, 2 were not written in English, and 1 was on an ongoing trial. By screening the references of the reviews, 3 additional articles were included in this study. Thus, a total of 42 articles met the criteria for the review. In terms of the CCA tools analyzed in these studies, they included five types: virtual reality (VR)-based, robot-based, telephone-based, smartphone-based, and computer-based cognitive assessments. Patients' stages of the disease ranged from the subacute phase and rehabilitation phase to the community phase. A total of 27 studies supported the effectiveness of CCA tools, while 22 out of 42 articles mentioned their benefits and 32 revealed areas for future improvement of CCA tools.
Conclusions: Although the use of CCA tools for assessing the cognition of post-stroke patients is becoming popular, there are still some limitations and challenges of using such tools in stroke survivors. More evidence is thus needed to verify the value and specific role of these tools in assessing the cognitive impairment of stroke patients.
1. Introduction
Stroke is now the leading cause of death and disability globally (1, 2). Post-stroke cognitive impairment (PSCI) is one of the most common complications of stroke. It can impair the patients' cognition in terms of their attention, executive function, memory, language, and visuospatial function, among others. Additionally, PSCI aggravates patients' movement disorders, affects the progress of their rehabilitation, and increases the rates of disability and mortality (3). The incidence of PSCI is as high as 80.97% among those who suffer a stroke (4). If PSCI cannot be effectively controlled, it can adversely affect mental functioning, and even lead to rapidly progressive dementia. In other words, PSCI places a severe burden on families and society.
PSCI is defined as a series of syndromes that meet the diagnostic criteria for cognitive impairment within 6 months after the clinical event of a stroke. In clinical practice, several popular neuropsychological tests have been used to assess cognitive function in association with PSCI, including Mini-Mental State Examination (MMSE), Montreal Cognitive Assessment Scale (MoCA), and the National Institute of Neurological Disorders and Stroke and the Canadian Stroke Network (NINDS-CSN) standardized test (5, 6). However, these traditional neuropsychological scales/tests are associated with certain limitations, such as being time-consuming and subjective (7). In recent years, the popular methods of cognitive assessment have gradually shifted from paper-and-pencil testing to computerized assessment, involving the use of computers, digital tablets, handheld devices, or other digital interfaces to collect data, scores or evaluate neurological dysfunctions. Sternin reported their 35-year experience of using computerized cognitive assessment (CCA). They claimed that CCA can save time, increase scoring accuracy, and be applied to large populations in a robust and efficient manner (8). However, to the best of our knowledge, there has been no systematic evaluation of CCA tools describing their advantages over conventional paper-and-pencil-based assessment.
Against this background, we conducted a systematic review to determine the benefits of using CCA tools in stroke survivors. We also intended to determine the challenges associated with applying such tools and how we can enhance our assessment strategies in stroke management. Specially, in this study, we aim to answer the following questions raised in previous reports: (1) How many types of CCA tools are available? (2) What are the benefits of using CCA tools? (3) What are the limitations of CCA tools? (4) How can CCA tools be improved for future applications?
2. Methods
2.1. Eligibility criteria
The criteria for the inclusion of studies in this research were as follows:
• Population: stroke patients with impairment of cognition or cognitive domains.
• Intervention: cognitive assessment with computerized tools.
• Comparison/analyses used: no specific requirements.
• Outcomes: cognitive domains involved, patient compliance, specificity and sensitivity, pros and cons of assessment, side effects, use of care services.
• Study type: clinical trials, analytical studies, primary research, studies on individuals with a clinical diagnosis of cognitive impairment after stroke.
• Publication date and language: limited to articles published from January 1st, 2010, to August 1st, 2022, in English.
Studies were excluded based on the following criteria:
• Study type: case reports, conferences, expert consensus, animal studies, abstracts, guidelines, comments, reviews that included a range of study designs or conditions unless they provided separate data for clinical trials with stroke survivors.
• Language: report written in a language other than English.
2.2. Information sources and search strategies
The publication data, study type and language filters corresponding to the eligibility criteria were used. Additionally, the following terms were included in different combinations (Supplementary Table S1): stroke, cerebral hemorrhage, hemorrhage, brain infarction, infarction, cerebral infarction; and cognition, metacognition, cognition disorders; and evaluation, assessment, mental status and dementia tests, neuropsychological tests; and compute*, intelligent*. These search terms were entered as any field (title, mesh, keyword, abstract, main text) in PubMed, Embase, Scopus, JAMA Network, Cochrane Library and PsycINFO for searches of reports published from January 1st, 2010, to August 1st, 2022. The complete search strategies for each database can be found in the Supplementary Table S1.
2.3. Study selection and data extraction
Two authors (QZ and JW) independently selected abstracts to retrieve according to the inclusion and exclusion criteria. If the eligibility criteria were met, the full text was obtained and read for further selection. Data were independently extracted by two authors (QZ and JW) using a predetermined data collection template that contained title, authors, institution, study design, year of publication, subjects, groups, stage of stroke, intervention name, involved cognitive domains, purpose of intervention, basic architecture, main characteristics, other assessment, advantages, disadvantages, PSCI diagnosis, outcomes, conclusion, and basic mechanism. In addition, a subsequent cross-check was performed to ensure the accuracy of study selection and data extraction. Inconsistencies were resolved through discussion or by asking the corresponding author for advice until a consensus was reached.
2.4. Assessment of risk of bias
Two authors (QZ and XF) independently assessed the risk of bias among the included studies by using the modified Jadad scale (9) for included randomized controlled trials, and using the Newcastle-Ottawa Scale (NOS) (10) for included case-control studies and cohort studies. They also examined cross-sectional studies using the 11-item checklist recommended by the Agency for Healthcare Research and Quality (AHRQ) (11). The above tools that were adapted for use in this study can be found in the Supplementary Methods 1–4. Based on these tools, the overall rating of the general methodological quality of each study would be reported as high, moderate, or low. To allow comparison of the study quality across different study types, a summary score of “low” (Jadad 1–3, NOS 0–4, AHRQ 0–3), “moderate” (NOS 5–6, AHRQ 4–7), or “high” (Jadad 4–7, NOS 7–9, AHRQ 8–11) was assigned. Concordant agreements were achieved through discussion by two researchers. If disagreements occurred, the corresponding author would be asked for advice to reach agreement.
3. Results
3.1. Study selection
A total of 8,697 titles and abstracts were acquired from six databases: 3,656 from PubMed, 2,780 from Scopus, 1,268 from Embase, 695 from JAMA Network, 221 from Cochrane Library, and 77 from PsycINFO. The remaining 8,697 titles, abstracts, and methods were initially screened, of which 74 potentially eligible articles were selected for further review. Of these 74 articles, 29 articles were not relevant to our research aim or we were unable to extract relevant information, 3 articles were reviews, 2 articles were not written in English, and 1 was on an ongoing trial. Besides, by screening the references of the identified reviews, 3 additional articles were included in our study. A total of 42 full-text articles (12–53) met the eligibility criteria for this systematic review (Figure 1).
Figure 1. Reference screening flow chart. A total of 8,697 titles and abstracts were acquired from six databases: 3,656 from PubMed, 2,780 from Scopus, 1,268 from Embase, 695 from JAMA Network, 221 from Cochrane Library, and 77 from PsycINFO. The remaining 8,697 titles, abstracts, and methods were initially screened, of which 74 potentially eligible articles were selected for further review. Of these 74 articles, 29 articles were not relevant to our research aim or we were unable to extract relevant information, three articles were reviews, two articles were not written in English, and one was on an ongoing trial. Besides, by screening the references of the identified reviews, three additional articles were included in our study. A total of 42 full-text articles met the eligibility criteria for this systematic review.
3.2. Risk of bias among the included studies
The modified Jadad scale was used to assess the risk of bias for 2 randomized controlled trials (Supplementary Table S2), while NOS was used for 2 case-control studies and 18 cohort studies (Supplementary Table S3). In addition, the risk of bias of 20 cross-sectional studies were assessed by AHRQ (Supplementary Table S3). Of the 42 studies in this review, 10 articles were of high quality, 27 articles were classed as of moderate quality, and the remaining 5 articles were of low quality.
3.3. Study characteristics and types of CCA tools
Stages of stroke can usually be divided into three phases: subacute phase (within 2 weeks), rehabilitation phase (2–8 weeks), and community phase (8 weeks or more). Of the 42 identified studies, four focused on stroke patients in the subacute phase, 20 studies involved those in the community phase, two targeted patients in the subacute and rehabilitation phase, five involved those in the rehabilitation phase and community phase, and six included stroke patients in all stages, while the phases of the stroke patients in five studies were unclear (Table 1).
Table 1. Typical characteristics of computerized cognitive assessment tools in articles.
According to the 2021 Chinese Expert Consensus on Post-Stroke Cognitive Impairment Management, the diagnosis of PSCI (no dementia) must be based on baseline cognitive decline and impairment of at least one cognitive domain, while instrumental activities of daily living can be normal or slightly impaired. In all 42 selected studies, assessments for reaching a diagnosis of cognitive impairment (CI) only involved MMSE, MoCA, or other neuropsychological tests or evaluation of quality of life (Table 1), which is not comprehensive.
The CCA tools reported in the 42 studies could be divided into five types: those involving virtual reality (VR)-based, robot-based, telephone-based, smartphone-based and computer-based cognitive assessments (Table 1).
3.3.1. VR-based cognitive assessment tools
Five studies reported the use of VR-based cognitive assessment tools performed in-person, including SeeMe Virtual Interactive Shopping environment (12), Cognitive Assessment for Aphasia App (C3A) (13, 14), Multitasking in the City Test (MCT) (15), and VR Wisc-R Mazes (16). Almost all of these tools focused on stroke patients involved in acute-setting inpatient rehabilitation and community-dwelling patients. They involved the evaluation of executive functions, attention, memory, and visuospatial skills (Table 1). In addition, Nir-Hadad et al. (12) and Carelli et al. (16) verified that the SeeMe Virtual Interactive Shopping environment and VR Wisc-R Mazes could be used to assess executive functions and visuospatial abilities in the daily activities of stroke patients, by comparing them with healthy participants. Meanwhile, Wall (13) and Jovanovski et al. (15) demonstrated the validity of C3A and MCT by comparing with the results obtained for normal subjects, as well as by comparing these approaches with standard pen-and-paper tests (Table 1).
3.3.2. Robot-based cognitive assessment tools
Two studies described the use of the robot-based cognitive assessment tool named KINARM Exoskeleton Robotic Evaluation (BKIN Technologies, Kingston, ON, Canada) performed in person (17, 18). Robotic technologies can provide neuropsychological tasks for assessing visuomotor and cognitive functions (e.g., spatial planning, working memory, visual processing) of stroke survivors living in the community (Table 1). Singh et al. (17) confirmed that KINARM Exoskeleton Robotic Evaluation was an effective computational model for examining visual search, by comparing the results obtained for stroke patients and those obtained for young and old healthy adults (Table 1).
3.3.3. Telephone-based cognitive assessment tools
Properly speaking, telephone-based assessment does not fall within the scope of computerized assessment. But we found that in three studies on remote cognitive assessment of stroke patients in the community, the modified telephone interview of cognitive status (TICSm) was used. Therefore, we did not omit it. The involved cognitive domains in TICSm were orientation, recent memory, delayed memory, attention, calculation, and language (19–21) (Table 1). Of these studies, Pendlebury et al. (21) proved that TICSm is a feasible and valid telephone-based method of testing stroke patients, not only by comparing such patients with a control group and compairing this approach with a standard neuropsychological test, but also by analyzing the sensitivity and specificity of TICSm (21) (Table 1).
3.3.4. Smartphone-based cognitive assessment tools
A cognitive rehabilitation mobile game named Neuro-World and the Korean Version of the MMSE using a smartphone were shown to be appropriate smartphone-based tools for detecting multi-domain impairment of stroke patients (22, 23). It was found that they can be used to assess cognitive functions including orientation, registration, attention, calculation, memory, language, and visuospatial abilities. Both of these approaches were confirmed to be useful given their good agreement with the results of standard assessments (22, 23) (Table 1).
3.3.5. Computer-based cognitive assessment tools
A total of 30 studies on the computer-based cognitive assessment of stroke patients were identified. Of these, 21 studies on cognitive assessment tools focusing on evaluating multi-domain or global cognition featured the following tools: Computerized Activity Measure for Post-Acute Care (AM-PAC) (24, 25), NeuroTrax™ (also known as MindStreams®) (26–29), NIH Toolbox Cognition Battery (NIHTB-CB) (30–32), CogState Battery (CogState Ltd., Australia) (33, 34), Seoul Computerized Neuropsychological Test (SCNT, Maxmedica Inc.) (35–37), computerized shape cancellation task (38, 39), computerized touchscreen cancellation, sustained attention and spatial working memory (40), computerized test of Attention Performance, Delis-Kaplan Executive Function System and Tower of London (41), Ryokansan touch panel-type screening test (42), Intelligent Cognitive Assessment System (ICAS) (43), and remote acquisition of neuropsychological data (RAND) (44). Meanwhile, nine studies reported tools for the assessment of cognition for a single domain, including computerized visual search task (45, 46), the PC test Amunet (NeuroScios GmbH, Austria) (47), computerized Wisconsin Card Sorting Test (48), computerized Iowa Gambling Task (IGT) (49), Tablet-based Symbol Digit Modalities Test (T-SDMT) (50), computer adaptive testing in Neuro-QOL (51), auditory test of variables of attention (TOVA) (52), and computer-based visuomotor task (53) (Table 1).
The finding showed that AM-PAC (24) and ICAS (43) were valid tools for screening the cognition of stroke patients, not only by comparing such patients with a control group and comparing these tools with a standard neuropsychological test, but also by analyzing these tools' sensitivity and specificity. Besides, NeuroTrax™ (26), NIH Toolbox (30, 32), computerized CogState Battery (33, 34), computerized visual search task (45), computerized IGT (49), T-SDMT (50), and TOVA (52) were confirmed to be reliable cognitive assessment tools by comparing their results for stroke patients with those for a control group and by comparing these approaches with a standard neuropsychological test. Meanwhile, computer adaptive testing in Neuro-QOL (51) was analyzed by comparing the results for stroke patients with those for a control group and by evaluating the sensitivity/ specificity of this approach. The validity of the remaining computer-based assessment tools was confirmed by comparing stroke patients with a control group (31, 37, 39, 41, 42, 46, 47) (Table 1).
Overall, although only three studies supplied evidence by performing comparisons with a control group/standard neuropsychological test, and by performing sensitivity/specificity analyses, a total of 27 studies (64.3%) had supported the effectiveness of CCA tools.
3.4. Benefits and limitations of CCA tools
Of the 42 included articles, 22 mentioned the benefits of CCA tools. As shown in Table 2, 31.8% of the articles clarified that CCA helped clinicians achieve remote cognitive assessment of patients; 27.3% demonstrated that CCA tools were easy for physicians and patients to use; and 18.2% asserted that CCA provided a dynamic assessment of cognitive function, minimized dependence on language skills (aphasia-friendly), and could be used in other clinical populations such as those suffering from Parkinson's disease and Alzheimer's disease. The advantages reported in 13.6% of the articles included reducing the assessment time, increasing scoring accuracy, simulating diverse life situations, requiring no special training, being more feasible than pen-and-paper tests, and a high level of participants' satisfaction. Besides, 9.1% of studies reported that CCA could be programmed with different languages and used at home. Some studies also clarified that CCA could be applied to large populations and implemented in institutions lacking expensive equipment (23). Besides, robot-based CCA could provide support for patients' limbs to reduce fatigue (18). Meanwhile, Durisko et al. (44) emphasized the feasibility of using RAND system for virtual home-based assessment without prior face-to-face contact between a participant and researcher. Wallmark et al. (52) suggested that the auditory versions of such tools could avoid the possible risk of epileptic seizures in patients exposed to flashing screens.
Table 2. Benefits and limitations of computerized cognitive assessment (CCA) tools in articles.
Among the 42 identified studies, 15 analyzed the limitations of CCA tools. As shown in Table 2, 9 out of these 15 studies (60%) excluded patients with severe language, cognitive, and functional deficits. Moreover, 20% of the articles on these studies reported that the results of CCA were influenced by previous experience using a computer, and patients easily made mistakes when they had rarely used a computer or tablet before the test. Additionally, 13% of the articles described patients becoming disorientated, including suffering discomfort such as short-term eye strain, or fatigue. Besides, it was shown that, for a novel computerized touch panel-type screening test named Ryokansan et al. (42), the use of the tool did not involve detailed manual-handling tasks. Furthermore, Sandel et al. (25) reported that the implementation of AM-PAC in clinical environments and the success of the project were influenced by instrumental, technological, operational, resource-related, and cultural factors. Park et al. (23) also found that remote assessment was associated with the possibility of errors when compared with conventional in-person assessment, and that special consideration should be made when interpreting scores for attention/calculation and visuospatial functions. Finally, Wallmark et al. (52) and Wall et al. (14) presented certain adverse outcomes in CCA, such as failing to react to correct stimuli and missing data.
3.5. Areas for future improvement of CCA studies
A total of 32 studies mentioned areas and strategies for the future development of CCA (Table 3). Overall, 59.4% of studies (19/32) mentioned that further studies should be performed on a larger sample of subjects. In addition, 21.9% of articles (7/32) reported that advanced research should include specific patient groups (e.g., those with severe cognitive impairment or aphasia), while 15.6% (5/32) suggested examining the connection between lesion location and task performance to draw conclusions about the influence of stroke topography. Another five articles (15.6%) advised that the same CCA task should be implemented on various technologies. Besides, 12.5% of articles introduced strategies including the use of more extensive cognitive examinations and the examination of longitudinal test performance to document the typical recovery of function. Moreover, three articles (9.4%) recommended identifying and reducing factors associated with patients' poor performance, such as fatigue, difficulty working under pressure, infection in the acute phase, lack of effort, use of medication, psychiatric history, litigation status, learning disability, hearing loss, employability and quality of life. Three other articles (9.4%) proposed examining acute recovery following stroke onset in further study. Additionally, 6.25% of articles proposed various directions for future work, such as combining CCA with a computational model, investigating the differing clinical demands and resources in various clinical settings, and identifying those with deficits in Cognitive Instrumental Activities of Daily Living (C-IADL) and Domain-specific Health-related Quality of Life (HRQoL) to achieve appropriate intervention, discharge planning, support, and follow-up. Singh et al. (17) also reported that, in future CCA studies, this approach should be combined with eye tracking. Moreover, Nir-Hadad et al. (12) considered examining the ecological validity by comparing performance in the virtual adapted shopping task with performance of a similar task in a supermarket of CCA as a future direction. Van Tuijl et al. (46) also recommended that more etiological research be performed on the specific cognitive consequences of stroke. Carelli et al. (16) reported the need for additional studies on issues relating to human-computer interaction in tests. Finally, Wall et al. (14) suggested exploring clinicians' user acceptance, investigating the different clinical demands and resource implications in varying clinical settings.
Table 3. Areas for future improvement of CCA tools.
4. Discussion
This systematic review summarizes the current evidence for the use of CCA tools in post-stroke patients. CCA tools related to stroke in the selected articles were based on VR, robots, telephones, smartphones, and computers. The studies presented here identified these five assessment methods, which would be considered for further research. Each selected study varied in its aim of assessment and measured outcomes, but increasing evidence supports the use of CCA as a clinical or rehabilitation tool. Of the 42 papers included in this study, 27 provided evidence for the effectiveness of CCA tools. Moreover, it was definitively concluded that TICSm, AM-PAC and ICAS are reliable and valid CCA tools for post-stroke patients.
Except for the benefits reported by Sternin et al. (8), 22 out of 42 articles expounded on the advantages of CCA tools. First, remote assessment, rather than in-person assessment, was assessed in seven studies (19–23, 25, 44). Remote CCA makes it possible to communicate with patients at home (22, 44) or elsewhere over the telephone or via an app on a smartphone, without physical contact. This would be critical for stroke patients and physicians in situation like the COVID-19 pandemic. Second, several studies considered the benefits of CCA such as the ease of use (15, 18, 33, 42–44), even aphasia-friendly (13, 14, 41, 43). Compared with traditional pen-and-paper cognitive tests, stroke patients preferred a CCA tool featuring two large external-response keys (“yes” and “no” buttons) (33). In addition, it was shown that touch panel-based screening tests could be easily understood and performed by patients due to the use of clear images and simple methods (42). Third, the CCA tools could be used to dynamically assess cognitive function (12, 22, 25, 53). Taking the AM-PAC assessment sessions as an example, these take only 7.9 min on average for data acquisition, and they can be used to track and assess patients' function in situation ranging from institutional to community settings (25). Besides, CCA tools can be used in other languages (26, 43), different situations (12, 14, 15), and clinical populations (17, 31, 43, 51). They may provide support for patients (18) and be associated with higher satisfaction (13, 14, 44) than standard pen-and-paper tests.
However, 15 out of 42 articles revealed limitations of CCA tools as follows. First, patients with severe language, cognitive, and functional deficits were excluded from the studies on CCA tools. Little evidence on the application of CCA tools in severe stroke patients was thus reported, which is a research gap that should be bridged in further work. Second, the results of CCA may be influenced by previous experience of using a computer (14, 16, 50). There is thus a need for some computer or CCA tool training before inclusion in clinical trials in this field, especially for the elderly. Third, several studies reported that stroke patients became disorientated (12, 16) or even experienced discomfort (12, 14) especially when they were immersed in a complex VR environment. Patients complained of short-term eye strain, fatigue, or slight discomfort in these studies. In addition, in contrast to pen-and-paper tests, CCA tools may suffer certain potential problems during their designing (42), setting (25), installation, and running (14, 23, 52). These factors must be considered in the future development and application of CCA tools.
Furthermore, future directions for studies on CCA tools were reported in 32 articles. The top three suggestions for future studies were to expand the sample size, to include stroke patients with severe functional impairment, and to examine the connection between the stroke lesion and CCA performance. Interestingly, besides future studies strictly within the medical field, some CCA studies with an interdisciplinary status (in the medical and engineering fields) were also proposed, such as those examining the same task on various technologies (12, 23, 25, 43, 50), combining CCA with computational models (17, 18) and eye tracking (17), and resolving issues associated with human-computer interactions during the tests (16). Because cognitive impairment after stroke interferes with the quality of life in stroke victims (54), 2 out of 42 studies recommended identifying deficits in the ability to perform activities of daily living (C-IADL, HRQoL) in stroke patients (24, 51) in order to diagnose PSCI accurately and comprehensively in future research.
Apart from the future developments listed in Table 3, we identified two future challenges for the development of CCA. One is how to design task-specific tools for PSCI patients. As suggested in the 2016 and 2021 Chinese Expert Consensus on Post-Stroke Cognitive Impairment Management (55), attention, executive function, memory, language ability and visuospatial ability are the five core cognitive domains affected by stroke (56). To accurately describe the cognitive status of a stroke patients, all of these five core domains need to be comprehensively assessed using CCA tools (57). However, we found that 10 of 42 articles focused on only one cognitive domain, while the others mainly assessed two or more cognitive domains. If a CCA included a task-specific assessment tool, it would remain an issue of how best to adjust the order or process of using CCA tools for testing different cognitive domains. To the best of our knowledge, to date, no research on this topic has been performed. The second challenge is how to improve the quality of clinical trials to study CCA tools. In this review, less than one-quater of the studies (10/42) were assessed as being of high quality, and only 2 of them involved randomized controlled trials. Thus, in future studies on CCA tools for PSCI patients, there is a need for high-quality clinical trials.
5. Conclusions
The studies identified in this review mainly focused on five types of CCA tools: those for VR-based, robot-based, telephone-based, smartphone-based, and computer-based cognitive assessments. Computer-based cognitive assessment tools were the most studied, but presented mixed results in the cognitive assessment of post-stroke patients. Certain limitations and challenges of using CCA tools in stroke survivors remain to be overcome, and more evidence is needed to verify the value and specific role of these tools in assessing the cognitive impairment of patients with stroke.
5.1. Limitations
Some limitations of this study should be presented. First, only two randomized controlled trials were included in this study, which affected the level of evidence of this study. Second, no deep data synthesis or statistical analysis was performed. Third, most of the identified studies still used MMSE or MoCA as a standard method to reveal the advantages or disadvantages of CCA tools, so further studies are needed to improve the objectivity and precision.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding authors.
Author contributions
X-QD and BZ contributed to the study conception and design. Material preparation, paper collection, and analysis were performed by QZ, J-HW, XF, and X-YL. The first draft of the manuscript was written by X-QD and XL. All authors commented on previous versions of the manuscript, read, and approved the final manuscript.
Funding
This work was supported by the Fundamental Research Funds from the Science and Technology Department of Jilin Province (grant number 20210101359JC, 2021) and Jilin University in China (grant number 2020B41, 2020).
Acknowledgments
Special thanks should go to BioMed Proofreading LLC in United States who helped with professional proofreading.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur.2023.1180664/full#supplementary-material
Abbreviations
PSCI, Post-stroke cognitive impairment; CCA, Computerized cognitive assessment; MMSE, Mini-mental state examination; MoCA, Montreal cognitive assessment scale; NOS, Newcastle-Ottawa Scale; AHRQ, Agency for Healthcare Research and Quality; TICSm, Modified Telephone Interview of Cognitive Status; AM-PAC, Activity Measure for Post-Acute Care; ICAS, Intelligent Cognitive Assessment System; RAND, Remote acquisition of neuropsychological data; IGT, Iowa Gambling Task; T-SDMT, Tablet-based Symbol Digit Modalities Test; TOVA, Test of variables of attention; CI, Cognitive impairment; VR, Virtual reality.
References
1. Xing L, Jing L, Tian Y, Wang W, Sun J, Jiang C, et al. Epidemiology of stroke in urban northeast China: A population-based study 2018-2019. Int J Stroke. (2021) 16:73–82. doi: 10.1177/1747493019897841
2. Pan Y, Zhang L, Li Z, Meng X, Wang Y, Li H, et al. Cost-effectiveness of a multifaceted quality improvement intervention for acute ischemic stroke in China. Stroke. (2020) 51:1265–71. doi: 10.1161/STROKEAHA.119.027980
3. Jokinen H, Melkas S, Ylikoski R, Pohjasvaara T, Kasta M, Erkinjuntti T, et al. Post-stroke cognitive impairment is common even after successful clinical recovery. Eur J Neurol. (2015) 22:1288–94. doi: 10.1111/ene.12743
4. Qu Y, Zhuo L, Li N, Hu Y, Chen W, Zhou Y, et al. Prevalence of post-stroke cognitive impairment in china: a community-based, cross-sectional study. PLoS ONE. (2015) 10:e0122864. doi: 10.1371/journal.pone.0122864
5. Mao L, Chen XH, Zhuang JH Li P, Xu YX, Zhao YC, et al. Relationship between beta-amyloid protein 1-42, thyroid hormone levels and the risk of cognitive impairment after ischemic stroke. World J Clin Cases. (2020) 8:76–87. doi: 10.12998/wjcc.v8.i1.76
6. Chen X, Wong A, Ye R, Xiao L, Wang Z, Lin Y, et al. Validation of NINDS-CSN neuropsychological battery for vascular cognitive impairment in Chinese stroke patients. BMC Neurol. (2015) 15:20. doi: 10.1186/s12883-015-0270-z
7. Cahn-Hidalgo D, Estes PW, Benabou R. Validity, reliability, and psychometric properties of a computerized, cognitive assessment test (Cognivue®). World J Psychiatry. (2020) 10:1–11. doi: 10.5498/wjp.v10.i1.1
8. Sternin A, Burns A, Owen AM. Thirty-five years of computerized cognitive assessment of aging-where are we now? Diagnostics. (2019) 9:114. doi: 10.3390/diagnostics9030114
9. Xing K, Huang G, Hua S, Xu G, Li M. Systematic review of randomized controlled trials on antibiotic treatment for osteomyelitis in diabetes. Diabet Med. (2019) 36:546–56. doi: 10.1111/dme.13935
10. Cha BH, Jang MJ, Lee SH. Alcohol consumption can reduce the risk of gallstone disease: a systematic review with a dose-response meta-analysis of case-control and cohort studies. Gut Liver. (2019) 13:114–31. doi: 10.5009/gnl18278
11. Meng J, Gao C, Tang C, Wang H, Tao Z. Prevalence of hypochondriac symptoms among health science students in China: a systematic review and meta-analysis. PLoS ONE. (2019) 14:e0222663. doi: 10.1371/journal.pone.0222663
12. Nir-Hadad SY, Weiss PL, Waizman A, Schwartz N, Kizony R, A. virtual shopping task for the assessment of executive functions: validity for people with stroke. Neuropsychol Rehabil. (2017) 27:808–33. doi: 10.1080/09602011.2015.1109523
13. Wall K. (2017) Assessing cognition post-stroke using virtual reality technology. [master's thesis].[Brisbane, IL: University of Queensland.
14. Wall KJ, Cumming TB, Koenig ST, Pelecanos AM, Copland DA. Using technology to overcome the language barrier: the Cognitive Assessment for Aphasia App. Disabil Rehabil. (2018) 40:1333–44. doi: 10.1080/09638288.2017.1294210
15. Jovanovski D, Zakzanis K, Ruttan L, Campbell Z, Erb S, Nussbaum D. Ecologically valid assessment of executive dysfunction using a novel virtual reality task in patients with acquired brain injury. Appl Neuropsychol Adult. (2012) 19:207–20. doi: 10.1080/09084282.2011.643956
16. Carelli L, Rusconi ML, Scarabelli C, Stampatori C, Mattioli F, Riva G. The transfer from survey (map-like) to route representations into virtual reality mazes: effect of age and cerebral lesion. J Neuroeng Rehabil. (2011) 8:6. doi: 10.1186/1743-0003-8-6
17. Singh T, Fridriksson J, Perry CM, Tryon SC, Ross A, Fritz S, et al. A novel computational model to probe visual search deficits during motor performance. J Neurophysiol. (2017) 117:79–92. doi: 10.1152/jn.00561.2016
18. Mostafavi SM, Scott S, Dukelow S, Mousavi P. Reduction of assessment time for stroke-related impairments using robotic evaluation. IEEE Trans Neural Syst Rehabil Eng Jul. (2017) 25:945–55. doi: 10.1109/TNSRE.2017.2669986
19. Huang Y, Yang S, Jia J. Factors related to long-term post-stroke cognitive impairment in young adult ischemic stroke. Med Sci Monit. (2015) 21:654–60. doi: 10.12659/MSM.892554
20. Biffi A, Bailey D, Anderson CD, Ayres AM, Gurol EM, Greenberg SM, et al. Risk factors associated with early vs. delayed dementia after intracerebral hemorrhage. JAMA Neurol. (2016) 73:969–76. doi: 10.1001/jamaneurol.2016.0955
21. Pendlebury ST, Welch SJ, Cuthbertson FC, Mariz J, Mehta Z, Rothwell PM. Telephone assessment of cognition after transient ischemic attack and stroke: modified telephone interview of cognitive status and telephone montreal cognitive assessment versus face-to-face montreal cognitive assessment and neuropsychological battery. Stroke. (2013) 44:227–9. doi: 10.1161/STROKEAHA.112.673384
22. Jung HT, Daneault JF, Lee H, Kim K, Kim B, Park S, et al. Remote assessment of cognitive impairment level based on serious mobile game performance: an initial proof of concept. IEEE J Biomed Health Inform. (2019) 23:1269–77. doi: 10.1109/JBHI.2019.2893897
23. Park HY, Jeon SS, Lee JY, Cho AR, Park JH. Korean version of the mini-mental state examination using smartphone: a validation study. Telemed J E Health. (2017) 23:815–21. doi: 10.1089/tmj.2016.0281
24. Toglia J, Askin G, Gerber LM, Taub MC, Mastrogiovanni AR, O'Dell MW. Association between 2 measures of cognitive instrumental activities of daily living and their relation to the montreal cognitive assessment in persons with stroke. Arch Phys Med Rehabil. (2017) 98:2280–7. doi: 10.1016/j.apmr.2017.04.007
25. Sandel ME, Jette AM, Appelman J, Terdiman J, Teselle M, Delmonico R, et al. Designing and implementing a system for tracking functional status after stroke: a feasibility study. PMR. (2013) 56:481–90. doi: 10.1016/j.pmrj.2012.09.579
26. Shopin L, Shenhar-Tsarfaty S, Ben Assayag E, Hallevi H, Korczyn AD, Bornstein NM, et al. Cognitive assessment in proximity to acute ischemic stroke/transient ischemic attack: comparison of the montreal cognitive assessment test and mindstreams computerized cognitive assessment battery. Dement Geriatr Cogn Disord. (2013) 36:36–42. doi: 10.1159/000350035
27. Kliper E, Ben Assayag E, Korczyn AD, Auriel E, Shopin L, Hallevi H, et al. Cognitive state following mild stroke: A matter of hippocampal mean diffusivity. Hippocampus. (2016) 26:161–9. doi: 10.1002/hipo.22500
28. Kliper E, Ben Assayag E, Tarrasch R, Artzi M, Korczyn AD, Shenhar-Tsarfaty S, et al. Cognitive state following stroke: the predominant role of preexisting white matter lesions. PLoS ONE. (2014) 9:e105461. doi: 10.1371/journal.pone.0105461
29. Boussi-Gross R, Golan H, Volkov O, Bechor Y, Hoofien D, Beeri MS, et al. Improvement of memory impairments in poststroke patients by hyperbaric oxygen therapy. Neuropsychology. (2015) 29:610–21. doi: 10.1037/neu0000149
30. Carlozzi NE, Tulsky DS, Wolf TJ, Goodnight S, Heaton RK, Casaletto KB, et al. Construct validity of the NIH Toolbox Cognition Battery in individuals with stroke. Rehabil Psychol. (2017) 62:443–54. doi: 10.1037/rep0000195
31. Tulsky DS, Holdnack JA, Cohen ML, Heaton RK, Carlozzi NE, Wong AWK, et al. Factor structure of the NIH Toolbox Cognition Battery in individuals with acquired brain injury. Rehabil Psychol. (2017) 62:435–42. doi: 10.1037/rep0000183
32. Nitsch KP, Casaletto KB, Carlozzi NE, Tulsky DS, Heinemann AW, Heaton RK. Uncorrected versus demographically-corrected scores on the NIH toolbox cognition battery in persons with traumatic brain injury and stroke. Rehabil Psychol. (2017) 62:485–95. doi: 10.1037/rep0000122
33. Cumming TB, Brodtmann A, Darby D, Bernhardt J. Cutting a long story short: reaction times in acute stroke are associated with longer term cognitive outcomes. J Neurol Sci. (2012) 322:102–6. doi: 10.1016/j.jns.2012.07.004
34. Cumming TB, Brodtmann A, Darby D, Bernhardt J. The importance of cognition to quality of life after stroke. J Psychosom Res. (2014) 77:374–9. doi: 10.1016/j.jpsychores.2014.08.009
35. Kim BR, Kim DY, Chun MH Yi JH, Kwon JS. Effect of repetitive transcranial magnetic stimulation on cognition and mood in stroke patients: a double-blind, sham-controlled trial. Am J Phys Med Rehabil. (2010) 89:362–8. doi: 10.1097/PHM.0b013e3181d8a5b1
36. Yun GJ, Chun MH, Kim BR. The effects of transcranial direct-current stimulation on cognition in stroke patients. J Stroke. (2015) 17:354–8. doi: 10.5853/jos.2015.17.3.354
37. Kim TW, Kim YH, Kim KH, Chang WH. White matter hyperintensities and cognitive dysfunction in patients with infratentorial stroke. Ann Rehabil Med. (2014) 38:620–7. doi: 10.5535/arm.2014.38.5.620
38. Ten Brink AF, Biesbroek JM, Kuijf HJ, der Stigchel SV, Oort Q, Visser-Meily JMA, et al. The right hemisphere is dominant in organization of visual search-A study in stroke patients. Behav Brain Res. (2016) 304:71–9. doi: 10.1016/j.bbr.2016.02.004
39. Ten Brink AF, Van der Stigchel S, Visser-Meily JM, Nijboer TC. You never know where you are going until you know where you have been: Disorganized search after stroke. J Neuropsychol. (2016) 10:256–75. doi: 10.1111/jnp.12068
40. Dalmaijer ES Li KMS, Gorgoraptis N, Leff AP, Cohen DL, Parton AD, et al. Randomised, double-blind, placebo-controlled crossover study of single-dose guanfacine in unilateral neglect following stroke. J Neurol Neurosurg Psychiatry. (2018) 89:593–8. doi: 10.1136/jnnp-2017-317338
41. Schumacher R, Halai AD, Lambon Ralph MA. Assessing and mapping language, attention and executive multidimensional deficits in stroke aphasia. Brain. (2019) 142:3202–16. doi: 10.1093/brain/awz258
42. Deguchi K, Kono S, Deguchi S, Morimoto N, Kurata T. lkeda Y, et al. A novel useful tool of computerized touch panel-type screening test for evaluating cognitive function of chronic ischemic stroke patients. J Stroke Cerebrovasc Dis. (2013) 22:e197–206. doi: 10.1016/j.jstrokecerebrovasdis.2012.11.011
43. Yip CK, Man DW. Validation of the intelligent cognitive assessment system (ICAS) for stroke survivors. Brain Inj. (2010) 24:1032–8. doi: 10.3109/02699052.2010.490514
44. Durisko C, McCue M, Doyle PJ, Dickey MW, Fiez JA. A flexible and integrated system for the remote acquisition of neuropsychological data in stroke research. Telemed J E Health. (2016) 22:1032–40. doi: 10.1089/tmj.2015.0235
45. Schendel K, Dronkers NF, Turken AU. Not just language: persisting lateralized visuospatial impairment after left hemisphere stroke. J Int Neuropsychol Soc. (2016) 22:695–704. doi: 10.1017/S1355617716000515
46. van Tuijl JH, van Raak EPM, van Oostenbrugge RJ, Aldenkamp AP, Rouhl RPW. Cognition and quality of life in patients with poststroke epilepsy: a case-control study. Epilepsy Behav. (2020) 104:106444. doi: 10.1016/j.yebeh.2019.106444
47. Wu YF, Wu WB, Liu QP, He WW, Ding H, Nedelska Z, et al. Presence of lacunar infarctions is associated with the spatial navigation impairment in patients with mild cognitive impairment: a DTI study. Oncotarget. (2016) 7:78310–9. doi: 10.18632/oncotarget.13409
48. Fernandez-Andujar M, Doornink F, Dacosta-Aguayo R, Soriano-Raya JJ, Miralbell J, Bargallo N, et al. Remote thalamic microstructural abnormalities related to cognitive function in ischemic stroke patients. Neuropsychology. (2014) 28:984–96. doi: 10.1037/neu0000087
49. Escartin G, Junque C, Juncadella M, Gabarros A, de Miquel MA, Rubio F. Decision-making impairment on the Iowa Gambling Task after endovascular coiling or neurosurgical clipping for ruptured anterior communicating artery aneurysm. Neuropsychology. (2012) 26:172–80. doi: 10.1037/a0024336
50. Tung LC Yu WH, Lin GH Yu TY, Wu CT, Tsai CY, et al. Development of a Tablet-based symbol digit modalities test for reliably assessing information processing speed in patients with stroke. Disabil Rehabil. (2016) 38:1952–60. doi: 10.3109/09638288.2015.1111438
51. Naidech AM, Beaumont JL, Berman M, Francis B, Liotta E, Maas MB, et al. Dichotomous “good outcome” indicates mobility more than cognitive or social quality of life. Crit Care Med. (2015) 43:1654–9. doi: 10.1097/CCM.0000000000001082
52. Wallmark S, Lundstrom E, Wikstrom J, Ronne-Engstrom E. Attention deficits after aneurysmal subarachnoid hemorrhage measured using the test of variables of attention. Stroke. (2015) 46:1374–6. doi: 10.1161/STROKEAHA.115.009092
53. Tippett WJ AL, Rizkalla MN, Sergio LE, Black SE. True functional ability of chronic stroke patients. J NeuroEngineering Rehabil. (2013) 10:20. doi: 10.1186/1743-0003-10-20
54. Park JH, Kim BJ, Bae HJ, Lee J, Lee J, Han MK, et al. Impact of post-stroke cognitive impairment with no dementia on health-related quality of life. J Stroke. (2013) 15:49–56. doi: 10.5853/jos.2013.15.1.49
55. Aam S, Einstad MS, Munthe-Kaas R, Lydersen S. lhle-Hansen H, Knapskog AB, et al. Post-stroke cognitive impairment-impact of follow-up time and stroke subtype on severity and cognitive profile: the nor-coast study. Front Neurol. (2020) 11:699. doi: 10.3389/fneur.2020.00699
56. Kleineidam L, Stark M, Riedel-Heller SG, Pabst A, Schmiedek F, Streit F, et al. The assessment of cognitive function in the German National Cohort (NAKO) - Associations of demographics and psychiatric symptoms with cognitive test performance. World J Biol Psychiatry. (2022) 8:1–15. doi: 10.1080/15622975.2021.2011408
Keywords: cognitive assessment screening instrument, stroke, cognitive impairments, computer-aided design, review
Citation: Zhang Q, Wei J-H, Fu X, Liu X, Li X-Y, Liu W, Liu Z-L, Duan X-Q and Zheng B (2023) Can we trust computers to assess the cognition of stroke patients? A systematic review. Front. Neurol. 14:1180664. doi: 10.3389/fneur.2023.1180664
Received: 09 March 2023; Accepted: 09 May 2023;
Published: 25 May 2023.
Edited by:
Donna Clark Tippett, Johns Hopkins University, United StatesReviewed by:
Ishan Ghai, Jacobs University Bremen, GermanyAlexandra Economou, National and Kapodistrian University of Athens, Greece
Copyright © 2023 Zhang, Wei, Fu, Liu, Li, Liu, Liu, Duan and Zheng. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Xiao-Qin Duan, MTUyMDQzMDk3NjkmI3gwMDA0MDsxNjMuY29t; Bin Zheng, YmluLnpoZW5nJiN4MDAwNDA7dWFsYmVydGEuY2E=
†These authors have contributed equally to this work and share first authorship
‡ORCID: Xiao-Qin Duan orcid.org/0000-0003-1078-2037
Bin Zheng orcid.org/0000-0003-3476-5936