Michelle H. S. Tosin1*
Glenn T. Stebbins1
Christopher G. Goetz1
Randi J. Hagerman2
David Hessl3Melissa A. Zolecki4
Peter K. Todd5,6Maureen A. Leehey7
Deborah A. Hall1- 1Department of Neurological Sciences, Rush University Medical Center, Chicago, IL, United States
- 2Department of Pediatrics and the MIND Institute, University of California, Davis School of Medicine, Sacramento, CA, United States
- 3Department of Psychiatry and Behavioral Sciences and the MIND Institute, University of California Davis School of Medicine, Sacramento, CA, United States
- 4National Fragile X Foundation, McLean, VA, United States
- 5University of Michigan, Ann Harbor, MI, United States
- 6Ann Arbor Veterans Administration Healthcare System, Ann Arbor, MI, United States
- 7University of Colorado School of Medicine, Aurora, CO, United States
Background: The original Fragile X-associated Tremor Ataxia Syndrome Rating Scale (FXTAS-RS) contained 61 items, some requiring modifications to better meet recommendations for patient-focused rating scale development.
Purpose: Provide initial validation of a revised version of the FXTAS-RS for motor signs.
Method: We conducted a two-phase mixed-method approach. In Phase 1, revision, we implemented a Delphi technique identifying pertinent domains/subdomains and developing items through expert consensus. In Phase 2, content validation, we conducted cognitive pretesting assessing comprehensibility, comprehensiveness, and relevance of items to FXTAS motor signs.
Results: After five rounds of Delphi panel and two rounds of cognitive pretesting, the revised version of the FXTAS-RS was established with 18 items covering five domains and 13 subdomains of motor signs. Cognitive pretesting revealed adequate content validity for the assessment of FXTAS motor signs.
Conclusion: The revised FXTAS-RS has been successfully validated for content and it is now ready for large-scale field validation.
Introduction
Fragile X-associated tremor/ataxia syndrome (FXTAS) is an uncommon, genetic, movement disorder with motor and non-motor elements (1–4). Although there are currently no proven symptomatic or disease modifying treatments for FXTAS, the prompt identification of affected individuals within fragile X families and an active network of fragile X researchers have advanced the field to early clinical trials (5–10). This shift to treatment prioritization has created the critical need to develop FXTAS outcome measures that are reliable, validated, and appropriate for use in Phases II and III clinical trials.
The first version of the FXTAS Rating Scale (FXTAS-RS) was developed in 2006, containing 61 items from other scales validated for similar movement disorders, such as ataxia, Huntington's disease, essential tremor, and Parkinson's disease, which were considered to approximate the signs presented by FXTAS patients (11). At the time of its creation, the items were extracted directly from their original scales, without a designated domain, such as parkinsonism, ataxia, tremor, etc. Furthermore, items were not allocated to domains, and wording and scaling of items were not modified to meet the FXTAS-specific signs. These decisions may explain at least in part how some of the items failed to reach recommended clinimetric standards when analyzed in a population of 295 videotaped FXTAS patients (12). The study results demonstrated that many of the items needed to be revised or dropped from the scale and subjected to clinimetric phases of cognitive pretesting and final validation through large-sample field testing (12). Thus, our aim was to develop and collect patients' input on a revised version of the FXTAS-RS designed to assess FXTAS motor signs.
Method
We conducted a two-phase mixed method approach using Delphi panel and cognitive pretesting techniques, following the Food and Drug Administration (FDA) guidance for instrument modification and content validation (13).
Phase I–FXTAS-RS revision
Between 2020 and 2021, nine specialists from five sites of the United States (US) participated in five rounds of Delphi panel discussions. There were five neurologists, a pediatrician with expertise in FXTAS, two psychologists, and two nurses, one of them also a patient advocate and a family representative of a patient with FXTAS.
In Round 1, specialists individually listed the domains and subdomains relevant for the assessment of the FXTAS motor signs, according to their definitions and taxonomy. In Round 2, they established consensus on the set of domains and subdomains previously listed, rating each item with regards to the importance of each item in assessing motor signs in FXTAS by choosing one of three response options: 1- The domain/subdomain is not essential, 2- The domain/subdomain is useful, but not essential, and 3- The domain/subdomain is essential. Data from their responses were analyzed using the content validity ratio (CVR) of ≥ 0.75, as determined by Lawshe Table (14, 15).
In Round 3, specialists assessed whether the 61 items composing the first version of FXTAS-RS were contained in the domains of interest identified in the previous round. Then, the specialists reviewed the items with weaknesses in the previous clinimetric analyses, establishing consensus on which ones should be maintained, improved, or excluded. Data from their responses were also analyzed using the CVR, and in-depth discussion occurred until the percentage of agreement of 80% was reached, or the disagreement was consensually justified. In Round 4, specialists established agreement on the most appropriate type of response options and anchors in terms of structure, forms of use, administration, and clinimetric analysis (16, 17). In Round 5, specialists followed the FDA guidance and reviewed the draft scale and voted on the modified version, providing suggestions for adequacy regarding the scale format, instructions for use, items (order and utterances) and response options (scaling and anchors) (13).
Phase II–content validation
Between January and February 2022, 10 stakeholders (five movement disorder specialists and five FXTAS patients) participated in cognitive pretesting activities. We included patients of either sex, ≥ 55 years of age who met diagnostic criteria for FXTAS (1, 18). We excluded medically unstable patients, those in an immediate postoperative period, or those with dementia clinically determined by the site investigator.
During the cognitive pretesting, the revised version of the FXTAS-RS was applied by a movement disorder neurologist to a patient with FXTAS and no demographic information was collected from these participants. Cognitive insights gained from participants' experiences using the FXTAS-RS were subjected to verbal protocol analysis (VPA) using the investigator's notes (19). VPA data were categorized according to comprehensibility, comprehensiveness, and relevance criteria for major or minor review. Minor revisions are edits made to the scale to address participant suggestions and comments that do not go back to another round of cognitive pretesting. Minor revisions include replacing words with synonyms, wording, punctuation, and/or word highlighting (underline, bold). Major revisions are those that require further review of the scale after revision. Major revisions include change of text structure (used to simplify writing), change in scaling and/or answer options, and/or replacement of words with change in meaning. Major and minor revisions were then incorporated into the revised version of the scale between rounds until data saturation was reached, defined by the absence of additional identified problems that required scale adjustments.
The original version of the FXTAS-RS was not presented to participants in the cognitive pretesting as a comparator.
Results
For all assignments posted in each round of the Delphi panel, members responded to the task with written or oral responses during web-based meetings. Adherence with tasks and subsequent attendance on web-based consensus meetings was 96.3%.
In Round 1, specialists listed six domains (tremor, parkinsonism, ataxia, eye movement disorders, dystonia, and neuropathy) and 65 subdomains of observable FXTAS motor signs considered relevant to FXTAS assessment. In Round 2, specialists established consensus for five domains (all the above except for neuropathy) and 13 subdomains.
In Round 3, 24 of the 61 items composing the first version of the FXTAS-RS were excluded because they assess the same subdomain; 13 were excluded due to clinimetric weaknesses; and six were not consistent with the agreed domains/subdomains. With a focused strategy to keep the scale practical and as short as possible without sacrificing completeness, specialists established consensus to retain 18 items of the original FXTAS-RS in the revised version, covering all domains/subdomains emerging from Round 2. In Round 4, specialists agreed on ordinal (Likert type) response options with text anchors for 16 items, with the majority (n = 13) having five response options (ranging from 0 to 4), and an item with 4, 6, and 9 response options ranging from 0 to 3, 0 to 5 and 0 to 8 each. For these items 0 means no involvement/impairment and the largest number corresponds to severe involvement/impairment. For two items, it was agreed that the most appropriate type of response option is dichotomous, where 0 means absence of signs and 1 means presence.
In Round 5, the scale was refined for clarity, the instructions for use were created, and the final revised version of FXTAS-RS was confirmed. The Delphi process and the results obtained in each of the rounds is represented in Figures 1, 2.
Figure 1. Domains, subdomains, items, and response options that reached consensus throughout the 5 rounds of the Delphi panel.
Figure 2. Domains, subdomains, items, and response options composing the second version of FXTAS-RS. *Subdomains represented by more than one domain. **Items represented by more than one subdomain.
For the cognitive pretesting, after two rounds no additional issues was identified, and data saturation was reached. The cognitive insights from this testing were incorporated into the revised version of the scale between rounds. All insights were categorized according to the comprehensibility criterion, as the revised version met the comprehensiveness and relevance criteria for all participants. In Round 1, of the 18 items of the FXTAS-RS, 12 received comments/suggestions from the participants, as well as the sections “Instructions for the Evaluator,” “Identification Header,” and “Ruler” (major review = 7, minor review = 13). In Round 2, of the 18 items of the FXTAS-RS, 10 received comments/suggestions from the participants, as well as the section “Instructions for the Evaluator,” and all required minor revision.
Based on provisional timing exercises in a small series of FXTAS subjects, (N = 5) the revised FXTAS-RS requires ~5 to 10 min to administer compared to ~20 to 30 min for the original 61-item version (See Supplemental materials for the revised FXTAS-RS).
Discussion
The process of developing a new clinical outcome assessment is arduous, as it requires planning, construction, evaluation, and validation phases (20). The planning phase is one of the most important, since from it, the instrument content and context of use, and severity descriptions and metrics are established. The present study, using a Delphi panel and cognitive pretesting, revised and established the construct validity by assessing the patient response to a revised version of the FXTAS-RS. This methodology followed a pre-specified plan that identified and established motor impairment domains and specific items with graded anchors relevant to FXTAS motor impairment.
FXTAS case studies and studies on the natural history of FXTAS have been published describing its clinical characteristics of motor disabilities (1, 2, 18, 21, 22). From this literature, the five domains established by the Delphi panel for the revised version of the scale were based on clinical and neuropathological evidence of cortical, subcortical, and cerebellar impairment.
As with many neurodegenerative diseases, the heterogeneity of FXTAS requires a motor scale that covers the wide range and severity of motor signs. Therefore, in addition to determining domains, it is important to ensure that specific characteristics will be captured by items corresponding to a wide range of subdomains. Although FXTAS-RS originated from the unification of items from selected scales used to access motor signs in other movement disorders, in the revised version the selected items were adapted to FXTAS context. The set of procedures implement in this study allowed for the development of a specified context of use and better content validity (and, eventually, better feasibility and precision) for using the scale in clinical and research contexts. This demonstrates that the revised version of the instrument meets the clinimetric recommendations for the development of clinical rating scales (20, 23, 24). The refinement also represents a practical time savings of several minutes, an advantage especially in patients with attentional, cognitive, or physical compromise.
This study has strengths, as the implementation of the Delphi panel and cognitive pretesting with quanti-qualitative approaches, recommended for conducting clinimetric research, was used both to plan, develop and validate the content of the improved version of the proposed instrument. Despite its strengths, this study had limitations, including the panel meetings were converted from in-person to virtual format with COVID restrictions perhaps leading to less effective communication during the Delphi meetings. In addition, the study phases were conducted entirely with North American participants. However, we understand that once the North American validation phase is completed, the revised version of the FXTAS-RS will be made available for translation and cross-cultural validation, allowing the scale to be used in other global contexts.
Conclusion
The first version of the FXTAS-RS was revised in the hopes of creating a modified more reliable and valid assessment of FXTAS motor signs. Future research will focus on the clinical validation of the revised version of the FXTAS-RS, with a goal launch in 2023.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving human participants were reviewed and approved by Institutional Review Board Institutional Review Board at Rush University Medical Center. ORA: 21082702-IRB01 Date IRB Approved: 9/20/2021. The patients/participants provided their written informed consent to participate in this study.
Author contributions
MT, GS, and DAH organized the database and the statistical analysis plan. MT wrote the first draft of the manuscript. All authors contributed to writing the manuscript, reviewing the statistical plan, approving the submitted version, and contributed to the conception and design of the study.
Funding
This study was funded by the Zivin Family Foundation, Steve, and Shirley Kaufman.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur.2022.977380/full#supplementary-material
References
1. Jacquemont S, Hagerman RJ, Leehey M, Grigsby J, Zhang L, Brunberg JA, et al. Fragile X premutation tremor/ataxia syndrome: molecular, clinical, and neuroimaging correlates. Am J Hum Genet. (2003) 72:869–78. doi: 10.1086/374321
2. Hagerman RJ, Leehey M, Heinrichs W, Tassone F, Wilson R, Hills J, et al. Intention tremor, parkinsonism, and generalized brain atrophy in male carriers of fragile X. Neurology. (2001) 57:127–30. doi: 10.1212/WNL.57.1.127
3. Hagerman RJ, Leavitt BR, Farzin F, Jacquemont S, Greco CM, Brunberg JA, et al. Fragile-X–Associated Tremor/Ataxia Syndrome (FXTAS) in females with the FMR1 premutation. Am J Hum Genet. (2004) 74:1051–6. doi: 10.1086/420700
4. Berry-Kravis E, Potanos K, Weinberg D, Zhou L, Goetz CG. Fragile X-associated tremor/ataxia syndrome in sisters related to X-inactivation. Ann Neurol. (2005) 57:144–7. doi: 10.1002/ana.20360
5. Berry-Kravis EM, Lindemann L, Jønch AE, Apostol G, Bear MF, Carpenter RL, et al. Drug development for neurodevelopmental disorders: lessons learned from fragile X syndrome. Nat Rev Drug Discov. (2018) 17:280–99. doi: 10.1038/nrd.2017.221
6. Hall DA, Berry-Kravis E, Hagerman RJ, Hagerman PJ, Rice CD, Leehey MA. Symptomatic treatment in the fragile X–associated tremor/ataxia syndrome. Mov Disord. (2006) 21:1741–4. doi: 10.1002/mds.21001
7. Seritan AL, Nguyen D V, Mu Y, Tassone F, Bourgeois JA, Schneider A, et al. Memantine for fragile X–associated tremor/ataxia syndrome. J Clin Psychiatry. (2014) 75:264–71. doi: 10.4088/JCP.13m08546
8. Wang JY, Trivedi AM, Carrillo NR, Yang J, Schneider A, Giulivi C, et al. Open-label allopregnanolone treatment of men with fragile X-associated tremor/ataxia syndrome. Neurotherapeutics. (2017) 14:1073–83. doi: 10.1007/s13311-017-0555-6
9. Napoli E, Schneider A, Wang JY, Trivedi A, Carrillo NR, Tassone F, et al. Allopregnanolone treatment improves plasma metabolomic profile associated with GABA metabolism in fragile X-associated tremor/ataxia syndrome: a pilot study. Mol Neurobiol. (2019) 56:3702–13. doi: 10.1007/s12035-018-1330-3
10. Yang JC, Rodriguez A, Royston A, Niu YQ, Avar M, Brill R, et al. Memantine improves attentional processes in fragile X-associated tremor/ataxia syndrome: electrophysiological evidence from a randomized controlled trial. Sci Rep. (2016) 6:21719. doi: 10.1038/srep21719
11. Leehey MA, Goetz CG, Berry-Kravis EM. Development of the FXTAS rating scale for quantitative motor analysis. In: Society IP and MD, editor. 10th International Congress of Parkinson's Disease and Movement Disorders. Kyoto, Japan: Movement Disorders (2006).
12. Hall DA, Stebbins GT, Jacquemont S, Berry-Kravis E, Goetz CG, Hagerman R, et al. Clinimetric properties of the fragile X-associated tremor ataxia syndrome rating scale. Mov Disord Clin Pract. (2019) 6:120–4. doi: 10.1002/mdc3.12708
13. US Food and Drug Administration. Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. (2009). p. 1–39.
14. Lawshe C. A quantitative approach to content validity. Pers Psychol. (1975) 28:563–57. doi: 10.1111/j.1744-6570.1975.tb01393.x
15. Haynes SN, Richard DCS, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Psychol Assess. (1995) 7:238–47. doi: 10.1037/1040-3590.7.3.238
16. DeCastellarnau A. A classification of response scale characteristics that affect data quality: a literature review. Qual Quant. (2018) 52:1523–59. doi: 10.1007/s11135-017-0533-4
17. Robinson MA. Using multi-item psychometric scales for research and practice in human resource management. Hum Resour Manage. (2018) 57:739–50. doi: 10.1002/hrm.21852
18. Hagerman PJ, Hagerman RJ. The fragile-X premutation: a maturing perspective. Am J Hum Genet. (2004) 74:805–16. doi: 10.1086/386296
19. Stanton NA, Salmon PM, Rafferty LA, Walker GH, Baber C, Jenkins DP. Human Factors Methods. 2nd Edn. London: CRC Press (2017). doi: 10.1201/9781315587394
20. Benson J, Clark F. A guide for instrument development and validation. Am J Occup Ther. (1982) 36:789–800. doi: 10.5014/ajot.36.12.789
21. Salcedo-Arellano MJ, Wang JY, McLennan YA, Doan M, Cabal-Herrera AM, Jimenez S, et al. Cerebral microbleeds in fragile X–associated tremor/ataxia syndrome. Mov Disord. (2021) 36:1935–43. doi: 10.1002/mds.28559
22. Leehey MA, Berry-Kravis E, Min S-J, Hall DA, Rice CD, Zhang L, et al. Progression of tremor and ataxia in male carriers of theFMR1 premutation. Mov Disord. (2007) 22:203–6. doi: 10.1002/mds.21252
23. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Heal. (2018) 6:1–18. doi: 10.3389/fpubh.2018.00149
Keywords: fragile X syndrome, fragile X tremor ataxia syndrome, FXTAS, outcome measures, severity of illness index
Citation: Tosin MHS, Stebbins GT, Goetz CG, Hagerman RJ, Hessl D, Zolecki MA, Todd PK, Leehey MA and Hall DA (2022) Fragile X-associated tremor ataxia syndrome rating scale: Revision and content validity using a mixed method approach. Front. Neurol. 13:977380. doi: 10.3389/fneur.2022.977380
Received: 24 July 2022; Accepted: 24 August 2022;
Published: 14 September 2022.
Edited by:
Genko Oyama, Juntendo University, JapanReviewed by:
Chin-Hsien Lin, National Taiwan University Hospital, TaiwanDejan Budimirovic, Johns Hopkins Medicine, United States
Copyright © 2022 Tosin, Stebbins, Goetz, Hagerman, Hessl, Zolecki, Todd, Leehey and Hall. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Michelle H. S. Tosin, michelle_tosin@rush.edu