The Italian telephone-based Verbal Fluency Battery (t-VFB): standardization and preliminary clinical usability evidence

Background: This study aimed at standardizing and providing preliminary evidence on the clinical usability of the Italian telephone-based Verbal Fluency Battery (t-VFB), which includes phonemic (t-PVF), semantic (t-SVF) and alternate (t-AVF) verbal fluency tasks.

Methods: Three-hundred and thirty-five Italian healthy participants (HPs; 140 males; age range = 18–96 years; education range = 4–23 years) and 27 individuals with neurodegenerative or cerebrovascular diseases were administered the t-VFB. Switch number and cluster size were computed via latent semantic analyses. HPs underwent the telephone-based Mental State Examination (MMSE) and Backward Digit Span (BDS). Construct validity, factorial structure, internal consistency, test-retest and inter-rater reliability and equivalence with the in-person Verbal Fluency tasks were assessed. Norms were derived via Equivalent Scores. Diagnostic accuracy against clinical populations was assessed.

Results: The majority of t-VFB scores correlated among each other and with the BDS, but not with the MMSE. Switch number correlated with t-PVF, t-SVF, t-AVF scores, whilst cluster size with the t-SVF and t-AVF scores only. The t-VFB was underpinned by a mono-component structure and was internally consistent (Cronbach’s α = 0.91). Test-retest (ICC = 0.69–0.95) and inter-rater reliability (ICC = 0.98–1) were optimal. Each t-VFB test was statistically equivalent to its in-person version (equivalence bounds yielding a p < 0.05). Education predicted all t-VFB scores, whereas age t-SVF and t-AVF scores and sex only some t-SVF scores. Diagnostic accuracy against clinical samples was optimal (AUC = 0.81–0.86).

Discussion: The t-VFB is a valid, reliable and normed telephone-based assessment tool for language and executive functioning, equivalent to the in-person version; results show promising evidence of its diagnostic accuracy in neurological populations.

Introduction

Verbal fluency tasks are renowned measures of both language (i.e., lexical retrieval) and executive functioning (i.e., executive control over selective attention, inhibition, set-shifting and self-monitoring). These tests usually require to produce, within a given time (usually of 60 s), as many words as possible starting with a given letter (phonemic verbal fluency, PFV) or belonging to a given category (semantic verbal fluency, SVF) (Whiteside et al., 2016; Aita et al., 2018). There is also an alternate verbal fluency test (AVF), which requires to generate words by continuously alternating between phonemically and semantically cued ones, thus assessing set-shifting abilities (Costa et al., 2014).

As not requiring either visual or physical supports, verbal fluency tasks have been widely adopted within telephone-based cognitive assessment (Rapp et al., 2012; Bunker et al., 2017; Marceaux et al., 2019), both as screening measures and within comprehensive neuropsychological batteries (Carlew et al., 2020).

Telephone-based cognitive assessment is relevant to clinical (e.g., for not-viable/-preferable access to clinics due to logistical, geographical, economical or health safety reasons; Christodoulou et al., 2016; Caze et al., 2020; De Cola et al., 2020; Soldati et al., 2021) and experimental telemedicine (e.g., population-based/epidemiological studies, decentralized clinical trials and prevention campaigns; Crooks et al., 2005; Yaari et al., 2006; Herr and Ankri, 2013). However, function-specific telephone-based tests often do not meet rigorous statistical-methodological standards, especially within the Italian context (Zanin et al., 2022). Moreover, so far, standardized telephone-based cognitive assessment tools in Italy are limited to screening tests for the assessment of global cognitive status (Aiello et al., 2022a,d,e).

Telephone-based adaptations of existing tests require ad hoc norms, psychometrics and diagnostics even if their administration and scoring are comparable to their in-person versions, since unknown sources of systematic error variance may affect scores due to the different administration setting (Parlar et al., 2020; Fox-Fuller et al., 2021; Postal et al., 2021; Zanin et al., 2022). More specifically, threats to the equivalence between in-person and telephone-based measures have been identified in a lack of control over the testing environment (e.g., due to sources of distractions or facilitations), technical issues (e.g., telephone line instability) and interpersonal aspects (e.g., the absence of a face-to-face interaction), all these factors being leading confounders especially in examinees with sensory-motor, cognitive or behavioral disorders (Martin-Khan et al., 2010; Binng et al., 2020; Booth et al., 2021; Hunter et al., 2021). Moreover, the lack of ad hoc standardizations has been highlighted also by clinicians as a barrier to remote cognitive assessment procedures (Rochette et al., 2021).

Such issues likewise apply to telephone-based verbal fluency tasks, which have been recognized as not completely comparable to in-person ones (Hunter et al., 2021). Nevertheless, no ad hoc standardizations of telephone-based verbal fluency tasks have been provided so far.

Given the above premises, the present study aimed at standardizing, for the Italian population, a telephone-based Verbal Fluency Battery (t-VFB) comprising PVF, SVF, and AVF tasks, as well as at preliminarily assessing its clinical usability in individuals with cerebrovascular or neurodegenerative diseases.

Materials and methods

Participants

Three-hundred and thirty-five Italian healthy participants (HPs) from different regions of Italy represented the normative sample. HPs were recruited through both Authors’ personal acquaintances and advertising at the University of Milano-Bicocca and University of Padova. HPs had no history of (1) neurological/psychiatric disorders, (2) active psychotropic medications, (3) uncompensated/severe general-medical conditions and (4) uncorrected hearing deficits.

Twenty-seven outpatients with neurological diseases were consecutively recruited at two neuropsychology services in Northern Italy. A clinical neurological diagnosis formulated by a neurologist constituted the inclusion criteria for clinical groups. Diagnoses were posed according to current diagnostic criteria and based on neurological, neuroradiological and neuropsychological examinations. Exclusion criteria for patients were: (1) severe medical-general conditions in the acute phase; (2) severe behavioral impairment that would have undermined compliance; (3) uncorrected hearing deficits. Ten participants had ischemic/hemorrhagic stroke causing unilateral, cortical or subcortical hemispheric lesion (5 with right-sided hemispheric lesion, and 5 with left-sided lesion). Six individuals presented with hypokinetic, extra-pyramidal disorders: 3 had Parkinson’s disease (Postuma et al., 2015), whereas 3 had atypical parkinsonisms (Levin et al., 2016). Six individuals had small vessel disease (Shi and Wardlaw, 2016). Four had a mixed, atrophic-vascular dementia (Zekry et al., 2002). Diagnoses of stroke, small vessel disease and mixed dementia were supported by computerized tomography or magnetic resonance imaging, whereas those of extra-pyramidal disorders by single-photon emission computerized tomography of the dopamine transporter. All patients underwent, for clinical purposes, a neuropsychological battery comprising measures of overall cognitive efficiency (Mini-Mental State Examination (Measso et al., 1993), attention (digit cancellation test, Spinnler and Tognoni, 1987; Trail-Making Test, Siciliano et al., 2019), executive functions (Frontal Assessment Battery Appollonio et al., 2005; Raven Colored Progressive Matrices Basso et al., 1987; phonemic and alternate verbal fluency Costa et al., 2014; Stroop test Caffarra et al., 2002), language (Token Test, Spinnler and Tognoni, 1987; Boston Naming Test, Goodglass et al., 1983; semantic verbal fluency, Costa et al., 2014), verbal and visuo-spatial memory (Babcock test, Novelli et al., 1986; forward/backward digit and Corsi span, Monaco et al., 2013) and visuo-spatial/constructional abilities (Clock Drawing Test, Caffarra et al., 2011; design copy, Spinnler and Tognoni, 1987). Additionally, stroke patients with a left-sided hemispheric lesion underwent a comprehensive language (Esame Neuropsicologico per l’Afasia; Capasso and Miceli, 2001) and limb apraxia assessment (ideomotor praxis, De Renzi et al., 1980; ideative praxis, De Renzi and Lucchelli, 1988), whereas in those with a right-sided damage extra-personal and personal hemispatial neglect was assessed (Bells test, Vallar et al., 1994; letter cancellation test, Vallar et al., 1994; Apple Cancellation Test, Mancuso et al., 2015; line bisection test, Nichelli et al., 1989; Fluff test, Cocchini et al., 2001; Comb and Razor test, Zoccolotti et al., 1992).

Medical history for all participants was collected through a semi-structured interview.

The study received ethical approval by the Committees of the University of Milano-Bicocca, Milano (ID: RM-2021-382, 19/02/2021), the University of Padova, Padua (ID: 4107, 19/02/2021) and IRCCS Istituto Auxologico Italiano, Milano (ID: 25C122, 18/05/2021). Informed consent was acquired from every participant. Data collection for HPs started in March 2021 and ended in January 2022, whereas that for clinical populations started in July 2021 and ended in January 2022.

Materials

The t-VFB comprises the telephone-based versions of 3 verbal fluency subtests: the t-PVF, t-SVF, and t-AVF subtests standardized by Costa et al.’s (2014), which were administered over the telephone. The t-PVF requires to generate as many words as possible beginning with letters “F,” “A,” and “S,” within a 60”-timespan each. The t-SVF require to generate semantic category exemplars: “Color,” “Animal,” and “Fruit” (60 s for each category). The t-AVF requires to continuously alternate letter-cued words with category-cued words as follows: “A/Color,” “F/Animal,” “S/Fruit”; again, each trial lasted 60 s. Following the administration procedure for the in-person version of Costa et al.’s (2014) battery, the three verbal fluency subtests were administered consecutively in the following order: t-PVF, t-SVF, and t-AVF; according to the original work, PVF and SVF tasks must precede the AVF subtest in order to calculate the Composite Shifting Index (CSI), i.e., a measure of the cost of shifting from phonemic/semantic (single-cued) to alternating (double-cued) fluency. As in the original work by Costa et al. (2014), participants were instructed not to produce proper nouns, place names, numbers or inflected words with the same suffix. For each trial, the number of words generated in 60 s was recorded. Performance score in each subtest is computed as the sum of the number of words generated in all trials belonging to that specific subtest. A t-CSI was also computed as follows: t-AVF/[(t-PVF + t-SVF)/2]. The t-CSI thus addresses the words generated in all three subtests, and it reflects the shifting cost for passing from the single fluency subtests to the alternate one.

The protocol of the t-VFB is reported in Supplementary Material 1. The t-VFB protocol was conceptualized and approved without reservation by an Author board comprising a neurologist (IA), six researchers with expertise in neuropsychology and psychometrics (SZ, SM, NB, ENA, VP, and LD).

To assess convergent validity, HPs were remotely administered the Italian telephone-based Mini-Mental State Examination (Itel-MMSE) (Metitieri et al., 2001; Aiello et al., 2022b) and a telephone version of the backward digit span (BDS) task (Monaco et al., 2013). The BDS encompasses both the longer sequence recalled, measuring working memory capacity (BDS-WM), and the total number of recalled sequences (BDS-T), measuring sustained attention (Pasotti et al., 2022).

Procedures

Before telephone-based testing, all participants underwent a detailed sound-check to ensure a good quality of the call, as well as and a brief training focused on those actions required for the executions of telephone-based tasks (Supplementary Material 2). For each participant, a caregiver/cohabitee was required to preliminarily ensure that no distractions or facilitations were present within the administration setting. The same person was asked to confirm the correctness of the address information provided by the participant during the spatial orientation task of the Itel-MMSE.

To test between-modality equivalence, a subgroup of 47 HPs were also administered in-person verbal fluency tasks either before (N = 23) or after (N = 24) a 48-h distance from telephonic testing, to rule out possible carry-over effects.

Eighteen HPs were re-tested with the t-VFB after 14 days from the baseline for test-retest reliability, whereas 27 protocols were simultaneously scored by two independent raters for inter-rater reliability.

All the above sub-samples for validity/reliability analyses were randomly selected from the whole normative sample.

Data were collected by either licensed or trainee psychologists who first underwent a thorough training.

Statistical analyses

SPSS 27 (IBM Corp., 2020), R 4.1.0¹ and jamovi 1.6.23² were adopted to analyze data.

Convergent validity was explored through either Pearson’s or Spearman correlations for normally and non-normally distributed data, respectively (skewness and kurtosis values < |1| and | 3|, respectively; Kim, 2013).

Intra-class correlations were adopted to examine test-retest and inter-rater reliability. Internal consistency and factorial structure were tested via Cronbach’s α and a Principal Component Analysis (PCA), respectively.

Verbal fluency performance relies on both language (i.e., lexical-semantic abilities) and executive functioning (Whiteside et al., 2016; Aita et al., 2018). According to Troyer et al. (1997), a valid measure of lexical-semantic integrity derived from verbal fluency is the number of circumscribable clusters of semantically-related words (cluster size), while a measure of executive functioning is the frequency of transitions between these clusters (number of switches). Accordingly, to provide further construct validity evidence, the number of switches and cluster size were computed relying on measures of semantic relatedness derived via a Latent Semantic Analysis, which is a natural language processing statistical model (Landauer and Dumais, 1997).

A two one-sided test (TOST) procedure for dependent samples (Lakens, 2017) was adopted for testing the equivalence between telephone-based and in-person verbal fluency tasks. The TOST procedure regards a between-mean effect size as equivalent to 0 if it falls within the upper and lower equivalence bounds and distances from them at p < 0.05.

Norms were derived through the Equivalent Score (ES) approach (Capitani and Laiacona, 2017; Aiello and Depaoli, 2022). Accordingly, clinical judgments were allowed by identifying outer and inner tolerance limits (oTL; iTL), as well as ES thresholds, on stepwise regression-adjusted scores. The ES scales allows to draw clinical judgments as follows: ES = 0 (adjusted scores ≤ oTL) → impaired; ES = 1→ borderline; ES = 2→ “low-end” normal; ES = 3/4→ normal.

Diagnostic accuracy of the t-VFB was tested via receiver-operating characteristics analyses by addressing scores of the whole clinical group against the whole normative sample.

Power analyses

The minimum sample size for reliability and validity analyses in HPs were set at N≈20 and N≈80, respectively, pursuantly to Hobart et al.’s (2012) recommendations on psychometric measurement in neurology.

According to qualitative guidelines delivered by Kyriazos (2018), a sample of 100 subjects was deemed as sufficient for running the PCA.

Sample size estimation for the TOST procedure, as run through the R package TOSTER³ (Lakens, 2017), yielded a minimum of N = 44 pairs with a 90% power, α = 0.05 and upper and lower equivalence bounds of -0.5 and 0.5, respectively.

For norm derivation, an estimated N = 287 was deemed sufficient to detect a small-to-medium effect size (f² = 0.05) with a 90% power and a type-I error rate of 5% within a multiple regression model (df_numerator = 3) through the R package pwr⁴.

The minimum sample sizes for the normative and clinical groups for diagnostic accuracy analyses were set at N = 19 and N = 190, respectively, in accordance with Obuchowski’s (2005) procedures for a single-test ROC analysis and via easyROC⁵, by addressing a case-control allocation ratio of 10, AUC = 0.7, 1-β = 0.9 and α = 0.05.

Results

Table 1 shows the stratification of the normative sample, and Table 2 summarizes HPs’ demographic data and test scores.

TABLE 1

Table 1. Sample stratification for age, education, and sex.

TABLE 2

Table 2. Demographic and cognitive data of the normative sample.

Acceptability rate was 100%.

Convergence against the other telephone-based measures (Itel-MMSE and BDS) is reported in Supplementary Table 1. At α_adjusted = 0.001, the vast majority of t-VFB measures were associated with BDS-T and BDS-WM scores; by contrast, the Itel-MMSE was only associated with the t-AVF and t-CSI, but not to t-PVF and t-SVF scores.

When putting into relation all t-VFB measures among each other, significant, moderate-to-high correlations were detected [0.33 ≤ r_s(335) ≤ 0.92; all ps < 0.001] among t-PVF, t-SVF, and t-AVF, whereas, as to the t-CSI, significant associations where only detected with t-AVF scores [r_s(335) ≥ 0.54; all ps < 0.001]. At α_adjusted = 0.006, the number of switches was strongly related with total t-PVF, t-SVF and t-AVF scores [0.71 ≤ r_s(138) ≤ 0.74; all ps < 0.001], whereas cluster size was related to the t-SVF and t-AVF [both r_s(138) = -0.32, p < 0.001], but not to t-PVF [r_s(138) = -0.16, p = 0.067]. The t-CSI was not associated with either the number switches or cluster size (p ≥ 0.009).

The PCA revealed a mono-component structure underpinning t-PVF, t-SVF, and t-AVF scores (59.7% of variance explained; loading range = 0.65–0.83), here named “linguistically mediated executive functioning.” Internal consistency for such scores was excellent (Cronbach’s α = 0.91).

Inter-rater reliability was excellent for all t-VFB total scores: t-PVF: ICC = 1; t-SVF: ICC = 0.98; t-AVF: ICC = 0.98. Test-retest reliability was optimal for the t-PVF (ICC = 0.95) and t-AVF (ICC = 0.89), and good for the t-SVF (ICC = 0.69).

The TOST procedure revealed across-modality equivalence for all verbal fluency total scores: PVF: telephone-based: 42.6 ± 11.2 vs. in-person 41.7 ± 11, t(46) = 0.63, p = 0.533, upper and lower equivalence bounds both yielding a p ≤ 0.01; SVF: telephone-based: 55.9 ± 11.6 vs. in-person 53.7 ± 11.7, t(46) = -1.45, p = 0.154, upper and lower equivalence bounds both yielding a p < 0.05; AVF: telephone-based: 43.5 ± 11.1 vs. in-person 43.5 ± 12, t(46) = -0.01, p = 0.988, upper and lower equivalence bounds both yielding a p ≤ 0.001; CSI: telephone-based:0.89 ± 0.17 vs. in-person 0.91 ± 0.2, t(46) = 0.68, p = 0.501, upper and lower equivalence bounds both yielding a p ≤ 0.01.

Education predicted all t-VFB measures (ps ≤ 0.029), age the t-SVF, t-AVF and t-CSI (p ≤ 0.001), whereas sex predicted only t-SVF-colors (p = 0.006) and t-SVF-fruits scores (p < 0.001), with females performing better than males. Adjustment equations, TLS and ES thresholds are reported in Table 3; adjustment grids are reported in Supplementary Tables 2–4, whereas an automated adjusted sheet in Supplementary Material 3.

TABLE 3

Table 3. Adjustment equations and Equivalent Scores of the telephone-based Verbal Fluency Battery (t-VFB).

Table 4 reports demographic data and test scores of individuals with neurological diseases. When comparing the whole clinical group against the normative sample, each t-VFB task yielded excellent accuracy: t-PVF: AUC = 0.81, SE = 0.05, CI 95% [0.71, 0.91]; t-SVF: AUC = 0.81, SE = 0.04, CI 95% [0.73, 0.9]; t-AVF: AUC = 0.86, SE = 0.03, CI 95% [0.8, 0.93].

TABLE 4

Table 4. Patients’ demographic, clinical, and cognitive data.

Discussion

The present study provides Italian practitioners and researchers with a standardized, telephone-based set of verbal fluency tests (t-VFB), along with clinical usability evidence in participants affected with neurological conditions of different etiologies. As providing modality-specific norms, psychometrics and diagnostics of verbal fluency tasks to be administered over the telephone, this work is unprecedented both within the Italian and international literature, enriching the range of telephone-based cognitive tests available in Italy (Aiello et al., 2022a,d,c) by meeting statistical-methodological requirements suggested for telephone-based tools-related standardizations (Zanin et al., 2022). In this last respect, it is remarkable that the present telephone-based version of verbal fluency tests showed to be statistically equivalent to the in-person versions by Costa et al. (2014), this supporting, within longitudinal assessments, a flexible use of the t-VFB in combination with the in-person version normed by Costa et al. (2014). However, this finding should not lead users to address, for the t-VFB, the same norms, psychometrics and diagnostics of in-person verbal fluency tests. Indeed, in order to avoid distortion in test scores, such information needs to be derived through ad hoc standardization studies specific to the telephonic modality, consistently to international guidelines on remote cognitive testing (Postal et al., 2021; Zanin et al., 2022). Users of the t-VFB could nonetheless safely compare, within longitudinal assessments, the ESs yielded from its administration to those yielded by the administration of Costa et al.’s (2014) battery, since the 5-points ES scale allows to compare score at different tests (Capitani and Laiacona, 2017; Aiello and Depaoli, 2022).

As coming with norms separately for each task and subtask, the t-VFB represents a flexible tool as to its administration. Moreover, the t-VFB presents with a solid factorial structure, optimal construct validity, internal consistency, test-retest, and inter-rater reliability. This work also provides promising, albeit preliminary, diagnostic accuracy evidence of the t-VFB in individuals with cerebrovascular and neurodegenerative diseases. The construct validity of the t-VFB is also supported by its association with renowned, fine-grained measures of both language- and executive-related verbal fluency underpinnings, i.e., the cluster size and the number of switches, respectively (Troyer et al., 1997). Indeed, all t-VFB measures were related to the number of switches, i.e., the executive component common to all verbal fluency tasks (Whiteside et al., 2016), whereas only the t-SVF and t-AVF, which rely more on language functions as compared to PVF, correlated with cluster size, that is a proper language measure reflecting the integrity of the lexical-semantic component (Shao et al., 2014; Aita et al., 2018).

Data on demographic effects are overall in line with those reported by Costa et al. (2014): t-SVF, t-AVF, and t-CSI scores were predicted by both age and education, whereas the t-PVF only by the latter. However, in Costa et al. (2014), sex predicted all PVF, SVF, and AVF scores, whilst, within the present work, it was only predictive of certain t-SVF subtest scores. This discrepancy is surprising, considering that the samples of both studies had a similar number of females and males, i.e., 56% of females in Costa et al. (2014), whereas 58% in the present study. Sex differences in healthy individuals on verbal fluency have been previously reported, although current evidence is still unclear and varied from report to report (Heister, 1982; Gauthier et al., 2009). Nevertheless, previous findings appear to sufficiently converge as to a slight female advantage at least for PVF (Capitani et al., 1998).

As to the limitations, it should be noted that the present findings about the clinical usability of the t-VFB refer to a small sample of participants with neurological diseases of different etiology. Future studies should extend and verify the present results in larger and homogeneous clinical cohorts featured by executive and/or language disorders that could influence verbal fluency performance (Suárez-González et al., 2021). Within such clinical usability studies, post-test probabilities (positive and negative predictive values and likelihood ratios), responsiveness and reliable change need also to be assessed (Aiello et al., 2022c). Moreover, equivalence of the t-VFB with its in-person version needs to be documented also in neurological populations.

In conclusions, the t-VFB shows to be a valid, reliable, diagnostically accurate and normed telephone-based cognitive tool for the assessment of linguistically mediated executive functions. The t-VFB is equivalent to in-person verbal fluency tasks and shows promising evidence of clinical usability in individuals with cerebrovascular or neurodegenerative diseases.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://osf.io/75nsd/.

Ethics statement

The studies involving human participants were reviewed and approved by the Ethics Committee of the University of Milano-Bicocca, Milano (ID: RM-2021-382, 19/02/2021), the University of Padova, Padua (ID: 4107, 19/02/2021) and IRCCS Istituto Auxologico Italiano, Milano (ID: 25C122, 18/05/2021). The patients/participants provided their written informed consent to participate in this study.

Author contributions

EA and NB: conceptualization. EA, AP, VP, LD, AC, TD, SZ, IA, SM, and NB: data curation. EA, AP, VP, LD, and CB: formal analysis. NB: funding acquisition. EA, VP, LD, and AC: Investigation. EA and CB: Methodology. SZ, IA, SM, and NB: re-sources. TD, SZ, IA, and SM: supervision. EA, AP, VP, LD, CB, and NB: writing–original draft. EA, AP, VP, LD, AC, CB, TD, SZ, IA, SM, and NB: writing–review and editing. All authors have read and agreed to the published version of the manuscript.

Funding

Research funded by the Italian Ministry of Health to NB. Open access publication fees have been covered by Istituto Auxologico Italiano.

Acknowledgments

We thank Drs. Tiziana Metitieri and Paolo Caimi for providing us with the permission to use within this study the Italian telephone-based Mini-Mental State Examination.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2022.963164/full#supplementary-material

Footnotes

1. ^ https://cran.r-project.org/
2. ^ https://www.jamovi.org/
3. ^ https://cran.r-project.org/web/packages/TOSTER/TOSTER.pdf
4. ^ https://cran.r-project.org/web/packages/pwr/pwr.pdf
5. ^ http://www.biosoft.hacettepe.edu.tr/easyROC/

References

Aiello, E. N., and Depaoli, E. G. (2022). Norms and standardizations in neuropsychology via equivalent scores: software solutions and practical guides. Neurol. Sci. 43, 961–966. doi: 10.1007/s10072-021-05374-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Aiello, E. N., Esposito, A., Giannone, I., Diana, L., Appollonio, I., and Bolognini, N. (2022a). Telephone Interview for Cognitive Status (TICS): Italian adaptation, psychometrics and diagnostics. Neurol. Sci. 43, 3071–3077. doi: 10.1007/s10072-021-05729-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Aiello, E. N., Esposito, A., Giannone, I., Diana, L., Woolley, S., Murphy, J., et al. (2022d). ALS Cognitive Behavioral Screen-Phone Version (ALS-CBS™-PhV): norms, psychometrics, and diagnostics in an Italian population sample. Neurol. Sci. 43, 2571–2578. doi: 10.1007/s10072-021-05636-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Aiello, E. N., Pucci, V., Diana, L., Niang, A., Preti, A. N., Delli Ponti, A., et al. (2022e). Telephone-based Frontal Assessment Battery (t-FAB): standardization for the Italian population and clinical usability in neurological diseases. Aging Clin. Exp. Res. 34, 1635–1644. doi: 10.1007/s40520-022-02155-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Aiello, E. N., Esposito, A., Pucci, V., Mondini, S., Bolognini, N., and Appollonio, I. (2022b). Italian telephone-based Mini-Mental State Examination (Itel-MMSE): item-level psychometric properties. Aging Clin. Exp. Res. 34, 1259–1265. doi: 10.1007/s40520-021-02041-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Aiello, E. N., Rimoldi, S., Bolognini, N., Appollonio, I., and Arcara, G. (2022c). Psychometrics and diagnostics of Italian cognitive screening tests: a systematic review. Neurol. Sci. 43, 821–845. doi: 10.1007/s10072-021-05683-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Aita, S. L., Beach, J. D., Taylor, S. E., Borgogna, N. C., Harrell, M. N., and Hill, B. D. (2018). Executive, language, or both? An examination of the construct validity of verbal fluency measures. Appl. Neuropsychol. 26, 441–451. doi: 10.1080/23279095.2018.1439830

PubMed Abstract | CrossRef Full Text | Google Scholar

Appollonio, I., Leone, M., Isella, V., Piamarta, F., Consoli, T., Villa, M. L., et al. (2005). The Frontal Assessment Battery (FAB): normative values in an Italian population sample. Neurol. Sci. 26, 108–116. doi: 10.1007/s10072-005-0443-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Basso, A., Capitani, E., and Laiacona, M. (1987). Raven’s coloured progressive matrices: normative values on 305 adult normal controls. Funct. Neurol. 2, 189–194.

Google Scholar

Binng, D., Splonskowski, M., and Jacova, C. (2020). Distance assessment for detecting cognitive impairment in older adults: a systematic review of psychometric evidence. Dement. Geriatr. Cogn. Disord. 49, 456–470. doi: 10.1159/000511945

PubMed Abstract | CrossRef Full Text | Google Scholar

Booth, T., Murray, A., and Muniz-Terrera, G. (2021). Are we measuring the same thing? Psychometric and research considerations when adopting new testing modes in the time of COVID-19. Alzheimers Dement. 17, 251–254. doi: 10.1002/alz.12197

PubMed Abstract | CrossRef Full Text | Google Scholar

Bunker, L., Hshieh, T. T., Wong, B., Schmitt, E. M., Travison, T., Yee, J., et al. (2017). The SAGES telephone neuropsychological battery: correlation with in-person measures. Int. J. Geriatr. Psychiatry 32, 991–999. doi: 10.1002/gps.4558

PubMed Abstract | CrossRef Full Text | Google Scholar

Caffarra, P., Gardini, S., Zonato, F., Concari, L., Dieci, F., Copelli, S., et al. (2011). Italian norms for the Freedman version of the Clock Drawing Test. J. Clin. Exp. Neuropsychol. 33, 982–988. doi: 10.1080/13803395.2011.589373

PubMed Abstract | CrossRef Full Text | Google Scholar

Caffarra, P., Vezzadini, G., Dieci, F., Zonato, F., and Venneri, A. (2002). Una versione abbreviata del test di Stroop: dati normativi nella popolazione italiana. Nuova Riv. Neurol. 12, 111–115.

Google Scholar

Capasso, R., and Miceli, G. (2001). Esame Neuropsicologico per l’Afasia. Berlin: Springer Science.

Google Scholar

Capitani, E., and Laiacona, M. (2017). Outer and inner tolerance limits: their usefulness for the construction of norms and the standardization of neuropsychological tests. Clin. Neuropsychol. 31, 1219–1230. doi: 10.1080/13854046.2017.1334830

PubMed Abstract | CrossRef Full Text | Google Scholar

Capitani, E., Laiacona, M., and Basso, A. (1998). Phonetically cued word-fluency, gender differences and aging: A reappraisal. Cortex 34, 779–783. doi: 10.1016/S0010-9452(08)70781-0

CrossRef Full Text | Google Scholar

Carlew, A. R., Fatima, H., Livingstone, J. R., Reese, C., Lacritz, L., Pendergrass, C., et al. (2020). Cognitive assessment via telephone: a scoping review of instruments. Arch. Clin. Neuropsychol. 35, 1215–1233. doi: 10.1093/arclin/acaa096

PubMed Abstract | CrossRef Full Text | Google Scholar

Caze, T., Dorsman, K. A., Carlew, A. R., Diaz, A., and Bailey, K. C. (2020). Can You Hear Me Now? Telephone-Based Teleneuropsychology Improves Utilization Rates in Underserved Populations. Arch. Clin. Neuropsychol. 35, 1234–1239. doi: 10.1093/arclin/acaa098

CrossRef Full Text | Google Scholar

Christodoulou, G., Gennings, C., Hupf, J., Factor-Litvak, P., Murphy, J., Goetz, R. R., et al. (2016). Telephone based cognitive-behavioral screening for frontotemporal changes in patients with amyotrophic lateral sclerosis (ALS). Amyotroph. Later. Scler. Frontotemp. Degenerat. 17, 482–488. doi: 10.3109/21678421.2016.1173703

PubMed Abstract | CrossRef Full Text | Google Scholar

Cocchini, G., Beschin, N., and Jehkonen, M. (2001). The Fluff Test: A simple task to assess body representation neglect. Neuropsychol. Rehabil. 11, 17–31. doi: 10.1080/09602010042000132

CrossRef Full Text | Google Scholar

Costa, A., Bagoj, E., Monaco, M., Zabberoni, S., De Rosa, S., Papantonio, A. M., et al. (2014). Standardization and normative data obtained in the Italian population for a new verbal fluency instrument, the phonemic/semantic alternate fluency test. Neurol. Sci. 35, 365–372. doi: 10.1007/s10072-013-1520-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Crooks, V. C., Clark, L., Petitti, D. B., Chui, H., and Chiu, V. (2005). Validation of multi-stage telephone-based identification of cognitive impairment and dementia. BMC Neurology 5:8. doi: 10.1186/1471-2377-5-8

PubMed Abstract | CrossRef Full Text | Google Scholar

De Cola, M. C., Triglia, G., Camera, M., Corallo, F., Di Cara, M., Bramanti, P., et al. (2020). Effect of neurological screening on early dementia detection in southern Italy. J. Int. Med. Res. 48:0300060520949763. doi: 10.1177/0300060520949763

PubMed Abstract | CrossRef Full Text | Google Scholar

De Renzi, E., and Lucchelli, F. (1988). Ideational apraxia. Brain 111, 1173–1185. doi: 10.1093/brain/111.5.1173

PubMed Abstract | CrossRef Full Text | Google Scholar

De Renzi, E., Motti, F., and Nichelli, P. (1980). Imitating gestures: a quantitative approach to ideomotor apraxia. Arch. Neurol. 37, 6–10. doi: 10.1001/archneur.1980.00500500036003

PubMed Abstract | CrossRef Full Text | Google Scholar

Fox-Fuller, J. T., Rizer, S., Andersen, S. L., and Sunderaraman, P. (2021). Survey Findings About the Experiences, Challenges, and Practical Advice/Solutions Regarding Teleneuropsychological Assessment in Adults. Arch. Clin. Neuropsychol. 37, 274–291. doi: 10.1093/arclin/acab076

CrossRef Full Text | Google Scholar

Gauthier, C. T., Duyme, M., Zanca, M., and Capron, C. (2009). Sex and performance level effects on brain activation during a verbal fluency task: a functional magnetic resonance imaging study. Cortex 45, 164–176. doi: 10.1016/j.cortex.2007.09.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Goodglass, H., Kaplan, E., and Weintraub, S. (1983). Boston naming test. Philadelphia, PA: Lea and Febiger.

Google Scholar

Heister, G. (1982). Sex differences in verbal fluency: a short note. Curr. Psychol. 2, 257–260. doi: 10.1007/BF03186768

CrossRef Full Text | Google Scholar

Herr, M., and Ankri, J. (2013). A critical review of the use of telephone tests to identify cognitive impairment in epidemiology and clinical research. J. Telemed. Telecare 19, 45–54. doi: 10.1177/1357633X12474962

PubMed Abstract | CrossRef Full Text | Google Scholar

Hobart, J. C., Cano, S. J., Warner, T. T., and Thompson, A. J. (2012). What sample sizes for reliability and validity studies in neurology? J. Neurol. 259, 2681–2694. doi: 10.1007/s00415-012-6570-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Hunter, M. B., Jenkins, N., Dolan, C., Pullen, H., Ritchie, C., and Muniz-Terrera, G. (2021). Reliability of telephone and videoconference methods of cognitive assessment in older adults with and without dementia. J. Alzheimers Dis. 81, 1625–1647. doi: 10.3233/JAD-210088

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, H. Y. (2013). Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis. Restorat. Dentist. Endodont. 38, 52–54. doi: 10.5395/rde.2013.38.1.52

PubMed Abstract | CrossRef Full Text | Google Scholar

Kyriazos, T. A. (2018). Applied psychometrics: sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology 9:2207. doi: 10.4236/psych.2018.98126

CrossRef Full Text | Google Scholar

Lakens, D. (2017). Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Soc. Psychol. Pers. Sci. 8, 355–362. doi: 10.1177/1948550617697177

PubMed Abstract | CrossRef Full Text | Google Scholar

Landauer, T. K., and Dumais, S. T. (1997). A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol. Rev. 104:211. doi: 10.1037/0033-295X.104.2.211

CrossRef Full Text | Google Scholar

Levin, J., Kurz, A., Arzberger, T., Giese, A., and Höglinger, G. U. (2016). The differential diagnosis and treatment of atypical parkinsonism. Dtsch. Ärztebl. Int. 113, 61–69. doi: 10.3238/arztebl.2016.0061

PubMed Abstract | CrossRef Full Text | Google Scholar

Mancuso, M., Rosadoni, S., Capitani, D., Bickerton, W. L., Humphreys, G. W., De Tanti, A., et al. (2015). Italian standardization of the apples cancellation test. Neurol. Sci. 36, 1233–1240. doi: 10.1007/s10072-015-2088-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Marceaux, J. C., Prosje, M. A., McClure, L. A., Kana, B., Crowe, M., Kissela, B., et al. (2019). Verbal fluency in a national sample: telephone administration methods. Int. J. Geriatr. Psychiatry 34, 578–587. doi: 10.1002/gps.5054

PubMed Abstract | CrossRef Full Text | Google Scholar

Martin-Khan, M., Wootton, R., and Gray, L. (2010). A systematic review of the reliability of screening for cognitive impairment in older adults by use of standardised assessment tools administered via the telephone. J. Telemed. Telecare 16, 422–428. doi: 10.1258/jtt.2010.100209

PubMed Abstract | CrossRef Full Text | Google Scholar

Measso, G., Cavarzeran, F., Zappalà, G., Lebowitz, B. D., Crook, T. H., Pirozzolo, F. J., et al. (1993). The mini-mental state examination: normative study of an Italian random sample. Dev. Neuropsychol. 9, 77–85. doi: 10.1080/87565649109540545

CrossRef Full Text | Google Scholar

Metitieri, T., Geroldi, C., Pezzini, A., Frisoni, G. B., Bianchetti, A., and Trabucchi, M. (2001). The Itel-MMSE: an Italian telephone version of the Mini-Mental State Examination. Int. J. Geriatr. Psychiatry. 16, 166–167. doi: 10.1002/1099-1166(200102)16:2<166::aid-gps290>3.0.co;2-m

CrossRef Full Text | Google Scholar

Monaco, M., Costa, A., Caltagirone, C., and Carlesimo, G. A. (2013). Forward and backward span for verbal and visuo-spatial data: standardization and normative data from an Italian adult population. Neurol. Sci. 34, 749–754. doi: 10.1007/s10072-012-1130-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Nichelli, P., Rinaldi, M., and Cubelli, R. (1989). Selective spatial attention and length representation in normal subjects and in patients with unilateral spatial neglect. Brain Cogn. 9, 57–70. doi: 10.1016/0278-2626(89)90044-4

CrossRef Full Text | Google Scholar

Novelli, G., Papagno, C., Capitani, E., and Laiacona, M. (1986). Tre test clinici di memoria verbale a lungo termine: taratura su soggetti normali. Arch. Psicol. Neurol. Psichiatr. 47, 278–296.

Google Scholar

Obuchowski, N. A. (2005). ROC analysis. Am. J. Roentgenol. 184, 364–372. doi: 10.2214/ajr.184.2.01840364

PubMed Abstract | CrossRef Full Text | Google Scholar

Parlar, M. E., Spilka, M. J., Wong Gonzalez, D., Ballantyne, E. C., Dool, C., Gojmerac, C., et al. (2020). “You can’t touch this”: delivery of inpatient neuropsychological assessment in the era of COVID-19 and beyond. Clin. Neuropsychol. 34, 1395–1410. doi: 10.1080/13854046.2020.1810324

PubMed Abstract | CrossRef Full Text | Google Scholar

Pasotti, F., De Luca, G., Aiello, E. N., Gramegna, C., Di Gangi, M., Foderaro, G., et al. (2022). A multi-component, adaptive Working Memory Assessment battery (WoMAB): validation and norms in an Italian population sample. Neurol. Sci. 43, 958–992. doi: 10.1007/s10072-021-05416-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Postal, K. S., Bilder, R. M., Lanca, M., Aase, D. M., Barisa, M., Holland, A. A., et al. (2021). Inter Organizational Practice Committee guidance/recommendation for models of care during the novel coronavirus pandemic. Arch. Clin. Neuropsychol. 36, 17–28. doi: 10.1093/arclin/acaa073

PubMed Abstract | CrossRef Full Text | Google Scholar

Postuma, R. B., Berg, D., Stern, M., Poewe, W., Olanow, C. W., Oertel, W., et al. (2015). MDS clinical diagnostic criteria for Parkinson’s disease. Move. Disord. 30, 1591–1601. doi: 10.1002/mds.26424

PubMed Abstract | CrossRef Full Text | Google Scholar

Rapp, S. R., Legault, C., Espeland, M. A., Resnick, S. M., Hogan, P. E., Coker, L. H., et al. (2012). Validation of a cognitive assessment battery administered over the telephone. J. Am. Geriatr. Soc. 60, 1616–1623. doi: 10.1111/j.1532-5415.2012.04111.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Rochette, A. D., Rahman-Filipiak, A., Spencer, R. J., Marshall, D., and Stelmokas, J. E. (2021). Teleneuropsychology practice survey during COVID-19 within the United States. Appl. Neuropsychol. doi: 10.1080/23279095.2021.1872576 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

Shao, Z., Janse, E., Visser, K., and Meyer, A. S. (2014). What do verbal fluency tasks measure? Predictors of verbal fluency performance in older adults. Front. Psychol. 5:772. doi: 10.3389/fpsyg.2014.00772

PubMed Abstract | CrossRef Full Text | Google Scholar

Shi, Y., and Wardlaw, J. M. (2016). Update on cerebral small vessel disease: a dynamic whole-brain disease. Stroke Vasc. Neurol. 1, 83–92. doi: 10.1136/svn-2016-000035

PubMed Abstract | CrossRef Full Text | Google Scholar

Siciliano, M., Chiorri, C., Battini, V., Sant’Elia, V., Altieri, M., Trojano, L., et al. (2019). Regression-based normative data and equivalent scores for Trail Making Test (TMT): an updated Italian normative study. Neurol. Sci. 40, 469–477. doi: 10.1007/s10072-018-3673-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Soldati, A. B., Almeida, C., Lima, M., Araujo, A., Araujo-Leite, M. A., and Silva, M. T. T. (2021). Telephone screening of cognitive status (TICS) in severe COVID-19 patients: Utility in the era of social isolation. Eneurologicalsci 22:100322. doi: 10.1016/j.ensci.2021.100322

PubMed Abstract | CrossRef Full Text | Google Scholar

Spinnler, H., and Tognoni, G. (1987). Standardizzazione e taratura italiana di test neuropsicologici. Neurological Sciences 6, 1–120.

Google Scholar

Suárez-González, A., Cassani, A., Gopalan, R., Stott, J., and Savage, S. (2021). When it is not primary progressive aphasia: a scoping review of spoken language impairment in other neurodegenerative dementias. Alzheimers Dement. 7:e12205. doi: 10.1002/trc2.12205

PubMed Abstract | CrossRef Full Text | Google Scholar

Troyer, A. K., Moscovitch, M., and Winocur, G. (1997). Clustering and switching as two components of verbal fluency: evidence from younger and older healthy adults. Neuropsychology 11, 138–146. doi: 10.1037//0894-4105.11.1.138

CrossRef Full Text | Google Scholar

Vallar, G., Rusconi, M. L., Fontana, S., and Musicco, M. (1994). Tre test di esplorazione visuo-spaziale: taratura su 212 soggetti normali. Arch. Psicol. Neurol. Psichiatria 55, 827–841.

Google Scholar

Whiteside, D. M., Kealey, T., Semla, M., Luu, H., Rice, L., Basso, M. R., et al. (2016). Verbal fluency: Language or executive function measure? Appl. Neuropsychol. 23, 29–34. doi: 10.1080/23279095.2015.1004574

PubMed Abstract | CrossRef Full Text | Google Scholar

Yaari, R., Fleisher, A. S., Gamst, A. C., Bagwell, V. P., and Thal, L. J. (2006). Utility of the telephone interview for cognitive status for enrollment in clinical trials. Alzheimers Dement. 2, 104–109. doi: 10.1016/j.jalz.2006.02.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Zanin, E., Aiello, E. N., Diana, L., Fusi, G., Bonato, M., Niang, A., et al. (2022). Tele-neuropsychological assessment tools in Italy: a systematic review on psychometric properties and usability. Neurol. Sci. 43, 125–138. doi: 10.1007/s10072-021-05719-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Zekry, D., Hauw, J. J., and Gold, G. (2002). Mixed dementia: epidemiology, diagnosis, and treatment. J. Am. Geriatr. Soc. 50, 1431–1143. doi: 10.1046/j.1532-5415.2002.50367.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Zoccolotti, P., Antonucci, G., and Judica, A. (1992). Psychometric characteristics of two semi-structured scales for the functional evaluation of hemi-inattention in extrapersonal and personal space. Neuropsychol. Rehabil. 2, 179–191. doi: 10.1080/09602019208401407

CrossRef Full Text | Google Scholar