D-KEFS ST Failure Identifies Multiple Sclerosis Patients With Worse Objective and Self-Perceived Physical and Cognitive Disability

Background and Objectives: The Brief Repeatable Battery of Neuropsychological Test (BRB-NT) does not explore the executive functions. We combined BRB-NT and Delis-Kaplan Executive Function System Sorting Test (D-KEFS ST) to obtain a more comprehensive evaluation of cognitive impairment in Multiple Sclerosis (MS) patients.

Methods: 137 Relapsing Remitting MS (RRMS) patients underwent a detailed neuropsychological assessment including BRB-NT, D-KEFS ST and self-administrated questionnaires, namely the Multiple Sclerosis Neuropsychological Questionnaire (MSNQ), the Fatigue Severity Scale (FSS) and the Beck Depression Inventory-Second Edition (BDI-II).

Results: Fifty-four patients (39.4%) had normal scores in each item of both batteries (cognitive normal), while 64 patients (46.7%) failed in at least one test of BRB-NT but not of D-KEFS ST (BRB-NT impaired) and 18 (13.1%) failed in at least one test of both batteries (BRB-NT+D-KEFS ST impaired). Only one patient (0.7%) failed in D-KEFS ST, but not in BRB-NT and was excluded from further analysis. BRB-NT+D-KEFS ST impaired patients had a significant higher mean disease duration and median EDSS score (15.5 ± 13.6 years and 3.5, respectively) compared to those with only BRB-NT impaired (7.9 ± 9.2, p < 0.01 and 2.5, p < 0.05) and with cognitive normal patients (6.7 ± 9.4, p < 0.005 and 2.0, p < 0.01). SDMT was more frequently impaired in BRB-NT+D-KEFS ST impaired patients (77.8%) compared to only BRB-NT impaired ones (20.0%, p < 0.001). The failure in D-KEFS ST was associated with the number of failed BRB-NT items (OR 1.46, IC95% 1.07–1.99, p < 0.05) and with pathological SDMT z-value (OR 10.56, IC95% 2.50–44.66, p < 0.005). Compared to BRB-NT impaired patients and the cognitive normal ones, BRB-NT+D-KEFS ST impaired patients had significant higher MSNQ (p < 0.01) and BDI-II (p < 0.05) values.

Conclusion: D-KEFS ST did not increase the number of cognitively impaired MS patients identified by BRB-NT, but provided a more comprehensive evaluation of cognitive decline. D-KEFS ST identified a subgroup of patients with increased self-perception of cognitive decline, depression and higher physical disability.

Introduction

Multiple Sclerosis (MS) is a chronic disabling neurological disorder affecting both the white and gray matter of the central nervous system. Up to 70% of MS patients develops various degrees of cognitive impairment that can be observed since early disease phases, sometimes at clinical onset (Chiaravalloti and DeLuca, 2008; Amato et al., 2010; Nourbakhsh et al., 2016).

Early cognitive impairment is considered a severe prognostic factor, tends to a progressive worsening and is not always linked to the accumulation of physical disability (Amato et al., 2006b; Moccia et al., 2016). Verbal and visuospatial memory, information processing speed and sustained attention are the most commonly compromised cognitive functions in MS. It is currently accepted that cognitively impaired patients are more likely to be unemployed or experiencing restrictions in social activities and household responsibilities (Rao et al., 1991a,b; Amato et al., 2001; Morrow et al., 2010; Ben Ari Shevil et al., 2014).

The most widely used tool to assess the cognitive profile in MS patients, the Brief Repeatable Battery of Neuropsychological Test (BRB-NT) (Rao, 1990), mainly explores memory, attention and information processing speed (with a sensitivity of 71% and a specificity of 94%), but does not properly investigate the executive functions. Indeed, about 17% of MS patients reveals failures in abstract and conceptual reasoning, fluency, planning and organization (Drew et al., 2008), thus limiting the ability of solving problems and making decisions.

To date, different tests have been applied to investigate various components of the executive domain in MS. Among these, (i) the Stroop Test (Stroop, 1935), that evaluates the ability to inhibit an automatic response while performing a task based on conflicting stimuli (Baysal Kıraç et al., 2014; Patti et al., 2015), (ii) the Tower of London (Shallice, 1982), that considers the ability to plan and to generate a sequence of goal-directed actions (Voelbel et al., 2011; Owens et al., 2013), and (iii) the Wisconsin Card Sorting Test (WCST) (Grant and Berg, 1948), that explores cognitive flexibility, problem solving and rule learning (van der Hiele et al., 2015; Koini et al., 2016).

Recently, the Delis-Kaplan Executive Function System (Delis et al., 2001) (D-KEFS) has been proposed for a more comprehensive evaluation of the executive functions. Indeed, D-KEFS has provided evidence for specific deficits in several neurological disorders, including traumatic brain injury, (Strong et al., 2011; Heled et al., 2012), Parkinson’s disease (McKinlay et al., 2010), dementia (Huey et al., 2009; Gansler et al., 2017), lateral prefrontal cortex lesions (Yochim et al., 2007), agenesis of the corpus callosum (Marco et al., 2012), stroke (Al-Dughmi et al., 2017), and amyotrophic lateral sclerosis (Libon et al., 2012). In particular, D-KEFS Sorting Test (D-KEFS ST), a subtest of this battery, was able to explore reasoning, categorization abilities, problem solving, abstraction, flexibility of thinking and concept-formation skills, providing a good validity (Parmenter et al., 2007) and an adequate reliability (Strauss et al., 2006). Despite WCST and D-KEFS ST gave comparable results in MS (Parmenter et al., 2007) relevant differences exist between these two tests. In particular, D-KEFS ST, but not WCST, discriminates the various components of the executive functions allowing the identification of specific impairments (Lezak, 2004). Moreover, D-KEFS ST investigates both verbal and non-verbal modalities of concept formation, does not employ the right/wrong feedback procedure (that could discouraged some individuals), and allows repeated measures, thus providing alternate forms. Recently, an abbreviate form of D-KEFS ST, i.e., the Card Sort 1, turned to be a good screening test for executive dysfunctions (Gromisch et al., 2016). Finally, an Italian study (Mattioli et al., 2014) provided normative data for D-KEFS ST. All these features might support D-KEFS ST as a compelling instrument to estimate an impairment of executive functions also in MS patients (Benedict et al., 2002).

However, it has to be pointed out that cognitive tests do not exhaustively evaluate all the neuropsychological aspects of MS. In order to obtain a more detailed view of patient’s cognition and perspective, self-evaluation questionnaires, mainly focused on perception of cognitive difficulties, fatigue and depression, are used in clinical practice. Namely, (i) fatigue, the so-called “invisible symptom” that interferes with cognitive, psychological and physical health, is commonly evaluated in MS patients by means of Fatigue Severity Scale (Nunnari et al., 2015; Ayache and Chalah, 2017; Lorefice et al., 2018); (ii) depression is evaluated with the Beck Depression Inventory-Second Edition (Siegert and Abernethy, 2005; Morrow et al., 2016; Patel and Feinstein, 2018); (iii) the Multiple Sclerosis Neuropsychological Questionnaire (MSNQ), a specific questionnaire that evaluates the perception of cognitive difficulties, is usually administrated, considering the frequent discrepancies between an objective assessment and the subjective viewpoint (Benedict et al., 2002; Julian et al., 2007; Kinsinger et al., 2010).

Taking into account all the above considerations, we tested to what extent D-KEFS ST could bring to light additional aspects of the cognitive impairment in MS not revealed by BRB-NT, but with potential relevant impact on patient’s daily life.

Materials and Methods

Patients

Hundred thirty-seven Relapsing-Remitting MS patients were enrolled in the study. At inclusion, the MS cohort had a mean age of 38.5 ± 10.9 years (range: 18–68), a mean disease duration of 8.4 ± 9.5 years (range: 0–42), a mean annualized relapse rate of 0.7 ± 0.5 (range: 0.0–2.0) and a median EDSS of 2.5 (range 0.0–7.0).

All the patients referred to the Multiple Sclerosis Centre of University Hospital of Padova for diagnosis (Polman et al., 2011) or clinical follow-up between March 2015 and April 2016. Exclusion criteria were history of other medical illness, learning disability, alcohol or drug abuse, impaired vision or hearing, and major psychiatric disorders.

The study was approved by the Ethic Committee of the Hospital of Padova. All the patients gave their written informed consent.

Neuropsychological and Neurological Evaluations

The neuropsychological evaluation consisted in the administration during the same assessment of BRB-NT and D-KEFS ST, in agreement with the original manuals and normative data for Italian population (Amato et al., 2006a; Goretti et al., 2014; Mattioli et al., 2014). BRB-NT included the Selective Reminding Test (Long Term Storage-LTS; Consistent Long Term Retrieval-CLTR; SRT-D), the Spatial Recall Test (SPART, SPART-D), the Symbol Digit Modalities Test (SDMT), the Paced Auditory Serial Addition Test (PASAT) and the Word List Generation (WLG). D-KEFS ST comprehended the Free Sorting Description (D-KEFS ST FSD) the Free Sorting Categorization (D-KEFS ST FSC), and the Sort Recognition (D-KEFS ST SR). Corrected values (for age, gender and education) and z-scores were considered in the analysis. The failure in one item was defined when the z-score was below 1.5 standard deviation.

Self-evaluation questionnaires, i.e., the Multiple Sclerosis Neuropsychological Questionnaire (MSNQ; Benedict et al., 2003) the Fatigue Severity Scale (FSS; Krupp et al., 1989) and the Beck Depression Inventory II (BDI-II; Beck et al., 1996) were also completed by all the patients.

The administration of the whole neuropsychological assessment took around 60 min.

The neurological examination was performed by trained neurologists (FR, MP, PP, and PG), specifying the EDSS score.

Statistics

ANOVA with Bonferroni’s correction was performed in order to evaluate normally distributed variables within the three patient’s subgroups identified (i.e., BRB-NT+D-KEFS ST cognitive impaired; BRB-NT impaired; cognitive normal). For qualitative ordinal variables the Mann-Whitney U-test was performed, while for qualitative nominal variables, the Pearson’s Chi square test was used. When all variables were normally distributed, correlations between variables were tested using the Pearson’s single or multiple linear models, while for ordinal variables Spearman correlation was applied. Finally, for dichotomic variables logistic regression analysis was also applied considering the dichotomic as dependent variable. The significance level was set at p < 0.05.

Results

BRB-NT+D-KEFS ST Impaired Patients Had Longer Disease Duration and Higher EDSS

Fifty-four patients (39.4%) had normal scores in each item of both batteries (cognitive normal), while 64 patients (46.7%) failed in at least one test of BRB-NT but not of D-KEFS ST (BRB-NT impaired) and 18 (13.1%) failed in at least one test of both batteries (BRB-NT+D-KEFS ST impaired). Only one patient (0.7%) failed in D-KEFS ST, but not in BRB-NT and was excluded from further analysis. Table 1 shows the main clinical and demographic variables of these groups. Cognitive normal patients were younger than BRB-NT impaired (p < 0.05) and BRB-NT+D-KEFS ST impaired patients (p < 0.005). BRB-NT+D-KEFS ST impaired patients had a longer disease duration compared to both cognitive normal (p < 0.005) and BRB-NT impaired patients (p < 0.01). Moreover, BRB-NT+D-KEFS ST impaired patients had a significant higher median EDSS score (3.5) compared to both BRB-NT impaired (2.5, p < 0.05) and cognitive normal patients (2.0, p < 0.01).

TABLE 1

Table 1. Study population: clinical and demographic features.

D-KEFS ST Is Predicted by BRB-NT Failure

BRB-NT+D-KEFS ST impaired patients had significant lower SDMT, PASAT and WLG z-values (-2.0 ± 1.1, -1.7 ± 0.8, and -0.9 ± 1.0, respectively) compared to BRB-NT impaired (-0.7 ± 1.1, p < 0.001, -0.9 ± 1.3, p < 0.05, and -0.2 ± 1.2 p < 0.05) and cognitive normal patients (0.4 ± 1.2, p < 0.001, 0.0 ± 1.0, p < 0.001, and -0.3 ± 1.0, p < 0.001) (Figure 1). SDMT, but not WLG or PASAT, was also more frequently impaired in BRB-NT+D-KEFS ST impaired patients (77.8%) than in BRB-NT impaired ones (18.8%, p < 0.001). All values and comparisons are in Supplementary Materials.

FIGURE 1

Figure 1. Cognitive normal, BRB-NT impaired and BRB-NT+D-KEFS ST impaired z-scores in BRB-NT items. While no difference was observed between cognitive normal and BRB-NT impaired, BRB-NT+D-KEFS ST impaired had significant lower SDMT and WLG z-values. ^∗p < 0.05, ^∗∗p < 0.01, ^∗∗∗p < 0.005, and ^∗∗∗∗p < 0.001.

Logistic regression analysis disclosed that the failure in D-KEFS ST (i.e., pathological z-score in at least one out of three subcomponents) was strongly associated with the number of failed BRB-NT items (OR 1.46, IC_95% 1.07–1.99, p < 0.05) and with pathological SDMT z-values (OR 10.56, IC_95% 2.50–44.66, p < 0.005). No association with other variables (especially, EDSS, disease duration and age) was disclosed. These findings were confirmed for each D-KEFS ST subtest (Table 2).

TABLE 2

Table 2. The impairment of D-KEFS ST Free Sorting Categorization (FSC), D-KEFS ST Free Sorting Description (FSD), and D-KEFS ST Sort Recognition (SR) associates with BRB-NT.

The Failure in D-KEFS ST Associates With Worse MSNQ and BDI-II Values

BRB-NT+D-KEFS ST impaired patients had significant higher MSNQ and BDI-II values (26.7 ± 14.4 and 14.4 ± 9.3, respectively) compared to BRB-NT impaired (17.3 ± 11.1, p < 0.01 and 9.3 ± 7.8, p < 0.05) and cognitive normal patients (16.1 ± 10.9, p < 0.01 and 8.3 ± 6.8, p < 0.05) (Figure 2). All values and comparisons are in Supplementary Materials.

FIGURE 2

Figure 2. Multiple Sclerosis Neuropsychological Questionnaire and BDI-II values in Cognitive normal, BRB-NT impaired and BRB-NT+D-KEFS ST impaired. While no difference was observed between Cognitive Normal and BRB-NT impaired, BRB-NT+D-KEFS ST impaired had significant higher MSNQ and BDI-II.

Discussion

We found that all MS patients failing D-KEFS ST also failed BRB-NT, indicating that BRB-NT is more sensitive than D-KEFS ST in recognize cognitive impairment in MS. Moreover, SDMT, one of the most sensitive tests to identify early cognitive decline in MS patients, was more frequently impaired and associated with executive dysfunctions observed by D-KEFS ST. In addition, as disclosed by logistic regression analysis, the number of failed BRB-NT items and the presence of impaired SDMT strongly associated with higher probability of a pathological D-KEFS ST z-score. Interestingly, WLG, the only test of BRB-NT that theoretically could explore a component of executive functions (namely, the categorization process) did not associate with D-KEFS ST. Taken all together, our data suggest that patients failing in SDMT have the highest risk of failure in executive functions and require a more detailed analysis of cognition, which should include D-KEFS ST.

On the other hand, the self-reported questionnaires analysis clearly indicated that depression and high perception of cognitive failures characterize exclusively MS patients failing D-KEFS ST. Although self-evaluation questionnaires are not considered an objective assessment, this finding might help clinicians to better understand the impact of disability on patient’s quality of life. Furthermore, since D-KEFS ST specifically identifies patients having more severe clinical (i.e., higher EDSS score) and neuropsychological impairments, this test may be useful to stratify MS subgroups for clinical studies.

A further relevant clinical rebound of our observations concerns the therapeutic management of the patients. It has been observed that the integrity of executive functions is associated with a better therapeutic compliance (Brock et al., 2011) and this aspect may be relevant in MS patients that manage drugs whose administration must be carefully scheduled, imply appropriate and specific clinical and laboratory follow-up and may present adverse events that must be quickly reported to clinicians. Thus, the presence of executive dysfunctions may significantly influence therapeutic decision-making (i.e., in favor of simpler therapeutic protocols) and should be explored in all MS patients that are initiating disease-modifying drugs. Whether D-KEFS ST may be used to predict therapeutic failures due to lack of adherence merits to be evaluated.

Finally, we are aware that the lack of a normal reference group might be considered a limit of our work. Nevertheless, we applied z-score analysis (validated for Italian population) (Amato et al., 2006a; Goretti et al., 2014; Mattioli et al., 2014) to weight correctly the observed differences.

Conclusion

In conclusion, although D-KEFS ST did not increase the number of cognitively impaired MS patients identified by BRB-NT, it provided a more comprehensive evaluation of the cognitive decline. In particular, D-KEFS ST failure identified a subgroup of patients with increased self-perception of cognitive dysfunctions, depression and higher physical disability. Whether these patients may be therapy-failures for lack of adherence merits further investigation in longitudinal studies.

Ethics Statement

The study was approved by the Local Ethic Committee. All the patients gave their informed consent.

Author Contributions

AR designed the study and collected the neuropsychological data. MP designed the study and performed the data analysis. FR and PP collected clinical data. ME performed data analysis. PG designed the study and contributed to the final version of the manuscript.

Conflict of Interest Statement

AR reports grants and personal fees from Novartis, grants and personal fees from Biogen Idec, grants from Teva, grants from Merck Serono, during the conduct of the study. MP reports grants and personal fees from Almirall, grants from Teva, grants and personal fees from Genzyme Sanofi, grants from Merck Serono, grants and personal fees from Biogen Idec, grants and personal fees from Novartis, during the conduct of the study. FR reports grants from Almirall, grants and personal fees from Teva, grants and personal fees from Genzyme Sanofi, grants and personal fees from Merck Serono, grants and personal fees from Biogen Idec, grants from Novartis, during the conduct of the study. ME has nothing to disclose. PP reports grants and personal fees from Merck Serono, grants and personal fees from Biogen Idec, grants and personal fees from Genzyme Sanofi, grants and personal fees from Bayer Schering Pharma, grants and personal fees from Novartis, grants and personal fees from Teva, during the conduct of the study. PG reports grants and personal fees from Merck Serono, grants and personal fees from Biogen Idec, grants and personal fees from Genzyme Sanofi, grants and personal fees from Bayer Schering Pharma, grants and personal fees from Novartis, grants and personal fees from Teva, grants from University of Padua, Department of Neurosciences DNS, grants from Veneto Region of Italy, grants from Italian Association for Multiple Sclerosis (AISM), grants from Italian Ministry of Public Health, during the conduct of the study.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00049/full#supplementary-material

References

Al-Dughmi, M., Al-Sharman, A., Stevens, S., and Siengsukon, C. F. (2017). Executive function is associated with off-line motor learning in people with chronic stroke. J. Neurol. Phys. Ther. 41:101106. doi: 10.1097/NPT.0000000000000170

PubMed Abstract | CrossRef Full Text | Google Scholar

Amato, M. P., Ponziani, G., Siracusa, G., and Sorbi, S. (2001). Cognitive dysfunction in early-onset multiple sclerosis. Arch. Neurol. 58:1602. doi: 10.1001/archneur.58.10.1602

CrossRef Full Text | Google Scholar

Amato, M. P., Portaccio, E., Goretti, B., Zipoli, V., Hakiki, B., Giannini, M., et al. (2010). Cognitive impairment in early stages of multiple sclerosis. Neurol. Sci. 31, 211–214. doi: 10.1007/s10072-010-0376-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Amato, M. P., Portaccio, E., Goretti, B., Zipoli, V., Ricchiuti, L., De Caro, M. F., et al. (2006a). The Rao’s brief repeatable battery and stroop test: normative values with age, education and gender corrections in an Italian population. Mult. Scler. J. 12, 787–793. doi: 10.1177/1352458506070933

PubMed Abstract | CrossRef Full Text | Google Scholar

Amato, M. P., Zipoli, V., Goretti, B., Portaccio, E., De Caro, M. F., Ricchiuti, L., et al. (2006b). Benign multiple sclerosis. J. Neurol. 253, 1054–1059. doi: 10.1007/s00415-006-0161-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Ayache, S. S., and Chalah, M. A. (2017). Fatigue in multiple sclerosis - Insights into evaluation and management. Neurophysiol. Clin. 47, 139–171. doi: 10.1016/j.neucli.2017.02.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Baysal Kıraç, L., Ekmekçi,Ö, Yüceyar, N., and Sağduyu Kocaman, A. (2014). Assessment of early cognitive impairment in patients with clinically isolated syndromes and multiple sclerosis. Behav. Neurol. 2014:637694. doi: 10.1155/2014/637694

PubMed Abstract | CrossRef Full Text | Google Scholar

Beck, A., Steer, R., and Brown, G. (1996). Beck Depression Inventory-II. Available at: http://m.blog.naver.com/mistyeyed73/220427762670

Google Scholar

Ben Ari Shevil, E., Johansson, S., Ytterberg, C., Bergström, J., and Von Koch, L. (2014). How are cognitive impairment, fatigue and signs of depression related to participation in daily life among persons with multiple sclerosis? Disabil. Rehabil. 36, 2012–2018. doi: 10.3109/09638288.2014.887797

PubMed Abstract | CrossRef Full Text | Google Scholar

Benedict, R. H., Munschauer, F., Linn, R., Miller, C., Murphy, E., Foley, F., et al. (2003). Screening for multiple sclerosis cognitive impairment using a self-administered 15-item questionnaire. Mult. Scler. J. 9, 95–101. doi: 10.1191/1352458503ms861oa

PubMed Abstract | CrossRef Full Text | Google Scholar

Benedict, R. H. B., Fischer, J. S., Beatty, W. W., Bobholz, J., Chelune, G. J., Langdon, D. W., et al. (2002). Minimal neuropsychological assessment of MS patients: a consensus approach. Clin. Neuropsychol. 16, 381–397. doi: 10.1076/clin.16.3.381.13859

PubMed Abstract | CrossRef Full Text | Google Scholar

Brock, L. L., Brock, C. D., and Thiedke, C. C. (2011). Executive function and medical non-adherence: a different perspective. Int. J. Psychiatry Med. 42, 105–115. doi: 10.2190/PM.42.2.a

PubMed Abstract | CrossRef Full Text | Google Scholar

Chiaravalloti, N. D., and DeLuca, J. (2008). Cognitive impairment in multiple sclerosis. Lancet Neurol. 7, 1139–1151. doi: 10.1016/S1474-4422(08)70259-X

CrossRef Full Text | Google Scholar

Delis, D., Kaplan, E., and Kramer, J. (2001). Delis-Kaplan Executive Function System (D-KEFS). San Antonio, TX: The Psychological Corporation.

Google Scholar

Drew, M., Tippett, L., Starkey, N., and Isler, R. (2008). Executive dysfunction and cognitive impairment in a large community-based sample with multiple sclerosis from New Zealand: a descriptive study. Arch. Clin. Neuropsychol. 23, 1–19. doi: 10.1016/j.acn.2007.09.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Gansler, D. A., Huey, E. D., Pan, J. J., Wasserman, E., and Grafman, J. H. (2017). Assessing the dysexecutive syndrome in dementia. J. Neurol. Neurosurg. Psychiatry 88, 254–261. doi: 10.1136/jnnp-2016-313576

PubMed Abstract | CrossRef Full Text | Google Scholar

Goretti, B., Patti, F., Cilia, S., Mattioli, F., Stampatori, C., Scarpazza, C., et al. (2014). The rao’s brief repeatable battery version B: normative values with age, education and gender corrections in an Italian population. Neurol. Sci. 35, 79–82. doi: 10.1007/s10072-013-1558-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Grant, D. A., and Berg, E. A. (1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. J. Exp. Psychol. 38, 404–411. doi: 10.1037/h0059831

PubMed Abstract | CrossRef Full Text | Google Scholar

Gromisch, E. S., Zemon, V., Holtzer, R., Chiaravalloti, N. D., DeLuca, J., Beier, M., et al. (2016). Assessing the criterion validity of four highly abbreviated measures from the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS). Clin. Neuropsychol. 30, 1032–1049. doi: 10.1080/13854046.2016.1189597

PubMed Abstract | CrossRef Full Text | Google Scholar

Heled, E., Hoofien, D., Margalit, D., Natovich, R., and Agranov, E. (2012). The delis–kaplan executive function system sorting test as an evaluative tool for executive functions after severe traumatic brain injury: a comparative study. J. Clin. Exp. Neuropsychol. 34, 151–159. doi: 10.1080/13803395.2011.625351

PubMed Abstract | CrossRef Full Text | Google Scholar

Huey, E. D., Goveia, E. N., Paviol, S., Pardini, M., Krueger, F., Zamboni, G., et al. (2009). Executive dysfunction in frontotemporal dementia and corticobasal syndrome. Neurology 72, 453–459. doi: 10.1212/01.wnl.0000341781.39164.26

PubMed Abstract | CrossRef Full Text | Google Scholar

Julian, L., Merluzzi, N. M., and Mohr, D. C. (2007). The relationship among depression, subjective cognitive impairment, and neuropsychological performance in multiple sclerosis. Mult. Scler. J. 13, 81–86. doi: 10.1177/1352458506070255

PubMed Abstract | CrossRef Full Text | Google Scholar

Kinsinger, S. W., Lattie, E., and Mohr, D. C. (2010). Relationship between depression, fatigue, subjective cognitive impairment, and objective neuropsychological functioning in patients with multiple sclerosis. Neuropsychology 24:573. doi: 10.1037/a0019222

PubMed Abstract | CrossRef Full Text | Google Scholar

Koini, M., Filippi, M., Rocca, M. A., Yousry, T., Ciccarelli, O., Tedeschi, G., et al. (2016). Correlates of executive functions in multiple sclerosis based on structural and functional mr imaging: insights from a multicenter study. Radiology 280, 869–879. doi: 10.1148/radiol.2016151809

PubMed Abstract | CrossRef Full Text | Google Scholar

Krupp, L. B., LaRocca, N. G., Muir-Nash, J., and Steinberg, A. D. (1989). The fatigue severity scale. Arch. Neurol. 46:1121. doi: 10.1001/archneur.1989.00520460115022

CrossRef Full Text | Google Scholar

Lezak, M. (2004). Neuropsychological Assessment, 4th Edn. New York, NY: Oxford University Press.

Google Scholar

Libon, D. J., McMillan, C., Avants, B., Boller, A., Morgan, B., Burkholder, L., et al. (2012). Deficits in concept formation in amyotrophic lateral sclerosis. Neuropsychology 26, 422–429. doi: 10.1037/a0028668

PubMed Abstract | CrossRef Full Text | Google Scholar

Lorefice, L., Fenu, G., Frau, J., Coghe, G. C., Marrosu, M. G., and Cocco, E. (2018). The impact of visible and invisible symptoms on employment status, work and social functioning in multiple sclerosis. Work 60, 263–270. doi: 10.3233/WOR-182682

PubMed Abstract | CrossRef Full Text | Google Scholar

Marco, E. J., Harrell, K. M., Brown, W. S., Hill, S. S., Jeremy, R. J., Kramer, J. H., et al. (2012). Processing speed delays contribute to executive function deficits in individuals with agenesis of the corpus callosum. J. Int. Neuropsychol. Soc. 18, 521–529. doi: 10.1017/S1355617712000045

PubMed Abstract | CrossRef Full Text | Google Scholar

Mattioli, F., Stampatori, C., Bellomi, F., Scarpazza, C., Galli, P., Guarneri, C., et al. (2014). Assessing executive function with the D-KEFS sorting test: normative data for a sample of the Italian adult population. Neurol. Sci. 35, 1895–1902. doi: 10.1007/s10072-014-1857-7

PubMed Abstract | CrossRef Full Text | Google Scholar

McKinlay, A., Grace, R. C., Dalrymple-Alford, J. C., and Roger, D. (2010). Characteristics of executive function impairment in Parkinson’s disease patients without dementia. J. Int. Neuropsychol. Soc. 16:268. doi: 10.1017/S1355617709991299

PubMed Abstract | CrossRef Full Text | Google Scholar

Moccia, M., Lanzillo, R., Palladino, R., Chang, K. C.-M., Costabile, T., Russo, C., et al. (2016). Cognitive impairment at diagnosis predicts 10-year multiple sclerosis progression. Mult. Scler. J. 22, 659–667. doi: 10.1177/1352458515599075

PubMed Abstract | CrossRef Full Text | Google Scholar

Morrow, S. A., Drake, A., Zivadinov, R., Munschauer, F., Weinstock-Guttman, B., and Benedict, R. H. B. (2010). Predicting loss of employment over three years in multiple sclerosis: clinically meaningful cognitive decline. Clin. Neuropsychol. 24, 1131–1145. doi: 10.1080/13854046.2010.511272

PubMed Abstract | CrossRef Full Text | Google Scholar

Morrow, S. A., Rosehart, H., and Pantazopoulos, K. (2016). Anxiety and depressive symptoms are associated with worse performance on objective cognitive tests in MS. J. Neuropsychiatry Clin. Neurosci. 28, 118–123. doi: 10.1176/appi.neuropsych.15070167

PubMed Abstract | CrossRef Full Text | Google Scholar

Nourbakhsh, B., Nunan-Saah, J., Maghzi, A.-H., Julian, L. J., Spain, R., Jin, C., et al. (2016). Longitudinal associations between MRI and cognitive changes in very early MS. Mult. Scler. Relat. Disord. 5, 47–52. doi: 10.1016/j.msard.2015.10.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Nunnari, D., De Cola, M. C., D’Aleo, G., Rifici, C., Russo, M., Sessa, E., et al. (2015). Impact of depression, fatigue, and global measure of cortical volume of cognitive impairment in multiple sclerosis. Biomed Res. Int. 2015:519785. doi: 10.1155/2015/519785

PubMed Abstract | CrossRef Full Text | Google Scholar

Owens, E. M., Denney, D. R., and Lynch, S. G. (2013). Difficulties in planning among patients with multiple sclerosis: a relative consequence of deficits in information processing speed. J. Int. Neuropsychol. Soc. 19, 613–620. doi: 10.1017/S1355617713000155

PubMed Abstract | CrossRef Full Text | Google Scholar

Parmenter, B. A., Zivadinov, R., Kerenyi, L., Gavett, R., Weinstock-Guttman, B., Dwyer, M. G., et al. (2007). Validity of the wisconsin card sorting and delis–kaplan executive function system (DKEFS) sorting tests in multiple sclerosis. J. Clin. Exp. Neuropsychol. 29, 215–223. doi: 10.1080/13803390600672163

PubMed Abstract | CrossRef Full Text | Google Scholar

Patel, V. P., and Feinstein, A. (2018). The link between depression and performance on the symbol digit modalities test: mechanisms and clinical significance. Mult. Scler. J. doi: 10.1177/1352458518770086 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

Patti, F., Nicoletti, A., Messina, S., Bruno, E., Fermo, S. L., Quattrocchi, G., et al. (2015). Prevalence and incidence of cognitive impairment in multiple sclerosis: a population-based survey in Catania, Sicily. J. Neurol. 262, 923–930. doi: 10.1007/s00415-015-7661-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Polman, C. H., Reingold, S. C., Banwell, B., Clanet, M., Cohen, J. A., Filippi, M., et al. (2011). Diagnostic criteria for multiple sclerosis: 2010 revisions to the mcdonald criteria. Ann. Neurol. 69, 292–302. doi: 10.1002/ana.22366

PubMed Abstract | CrossRef Full Text | Google Scholar

Rao, S. (1990). A Manual for the Brief Repeatable Battery of Neuropsychological Tests in Multiple Sclerosis. Milwaukee, WI: Medical College of Wisconsin.

Google Scholar

Rao, S. M., Leo, G. J., Bernardin, L., and Unverzagt, F. (1991a). Cognitive dysfunction in multiple sclerosis. I. Frequency, patterns, and prediction. Neurology 41, 685–691.

PubMed Abstract | Google Scholar

Rao, S. M., Leo, G. J., Ellington, L., Nauertz, T., Bernardin, L., and Unverzagt, F. (1991b). Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology 41, 692–696.

PubMed Abstract | Google Scholar

Shallice, T. (1982). Specific impairments of planning. Philos. Trans. R. Soc. Lond. B Biol. Sci. 298, 199–209. doi: 10.1098/rstb.1982.0082

PubMed Abstract | CrossRef Full Text | Google Scholar

Siegert, R. J., and Abernethy, D. A. (2005). Depression in multiple sclerosis: a review. J. Neurol. Neurosurg. Psychiatry 76, 469–475. doi: 10.1136/jnnp.2004.054635

PubMed Abstract | CrossRef Full Text | Google Scholar

Strauss, E., Sherman, E. M. S., Spreen, O., and Spreen, O. (2006). A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary. Oxford: Oxford University Press.

Google Scholar

Strong, C.-A. H., Tiesma, D., and Donders, J. (2011). Criterion validity of the delis-kaplan executive function system (D-KEFS) fluency subtests after traumatic brain injury. J. Int. Neuropsychol. Soc. 17, 230–237. doi: 10.1017/S1355617710001451

PubMed Abstract | CrossRef Full Text | Google Scholar

Stroop, J. R. (1935). Studies of interference in serial verbal reactions. J. Exp. Psychol. 18, 643–662. doi: 10.1037/h0054651

CrossRef Full Text | Google Scholar

van der Hiele, K., van Gorp, D., Ruimschotel, R., Kamminga, N., Visser, L., and Middelkoop, H. (2015). Work participation and executive abilities in patients with relapsing-remitting multiple sclerosis. PLoS One. 10:e0129228. doi: 10.1371/journal.pone.0129228

PubMed Abstract | CrossRef Full Text | Google Scholar

Voelbel, G. T., Goverover, Y., Gaudino, E. A., Moore, N. B., Chiaravalloti, N., and DeLuca, J. (2011). The relationship between neurocognitive behavior of executive functions and the EFPT in individuals with multiple sclerosis. OTJR 31, S30–S37. doi: 10.3928/15394492-20101108-06

PubMed Abstract | CrossRef Full Text | Google Scholar

Yochim, B., Baldo, J., Nelson, A., and Delis, D. C. (2007). D-KEFS trail making test performance in patients with lateral prefrontal cortex lesions. J. Int. Neuropsychol. Soc. 13, 704–709. doi: 10.1017/S1355617707070907

PubMed Abstract | CrossRef Full Text | Google Scholar