Callie E. Tyner1*
Pamela A. Kisala1
Aaron J. Boulton1
Mark Sherer2,3
Nancy D. Chiaravalloti4,5
Angelle M. Sander2,3Tamara Bushnik6,7David S. Tulsky1,8,9
- 1Center for Health Assessment Research and Translation, University of Delaware, Newark, DE, United States
- 2TIRR Memorial Hermann Research Center, Houston, TX, United States
- 3Baylor College of Medicine, Houston, TX, United States
- 4Kessler Foundation, East Hanover, NJ, United States
- 5Department of Physical Medicine and Rehabilitation, Rutgers New Jersey Medical School, Newark, NJ, United States
- 6NYU School of Medicine, New York, NY, United States
- 7NYU Langone Medical Center, New York, NY, United States
- 8Department of Physical Therapy, University of Delaware, Newark, DE, United States
- 9Department of Psychological and Brain Sciences, University of Delaware, Newark, DE, United States
Patient report of functioning is one component of the neurocognitive exam following traumatic brain injury, and standardized patient-reported outcomes measures are useful to track outcomes during rehabilitation. The Traumatic Brain Injury Quality of Life measurement system (TBI-QOL) is a TBI-specific extension of the PROMIS and Neuro-QoL measurement systems that includes 20 item banks across physical, emotional, social, and cognitive domains. Previous research has evaluated the responsiveness of the TBI-QOL measures in community-dwelling individuals and found clinically important change over a 6-month assessment interval in a sample of individuals who were on average 5 years post-injury. In the present study, we report on the responsiveness of the TBI-QOL Cognition–General Concerns and Executive Function item bank scores and the Cognitive Health Composite scores in a recently injured sample over a 1-year study period. Data from 128 participants with complicated mild, moderate, or severe TBI within the previous 6 months were evaluated. The majority of the sample was male, white, and non-Hispanic. The participants were 18–92 years of age and were first evaluated from 0 to 5 months post-injury. Eighty participants completed the 1-year follow-up assessment. Results show acceptable standard response mean values (0.47–0.51) for all measures and minimal detectable change values ranging from 8.2 to 8.8 T-score points for Cognition–General Concerns and Executive Functioning measures. Anchor rating analysis revealed that changes in scores on the Executive Function item bank and the Cognitive Health Composite were meaningfully associated with participant-reported changes in the areas of attention, multitasking, and memory. Evaluation of change score differences by a variety of clinical indicators demonstrated a small but significant difference in the three TBI-QOL change scores by TBI injury severity grouping. These results support the responsiveness of the TBI-QOL cognition measures in newly injured individuals and provides information on the minimal important differences for the TBI-QOL cognition measures, which can be used for score interpretation by clinicians and researchers seeking patient-reported outcome measures of self-reported cognitive QOL after TBI.
Introduction
Cognitive concerns are common after traumatic brain injury (TBI), although the presentation of subjective complaints and objective impairments are heterogeneous across individuals (Rabinowitz and Levin, 2014). Post-injury assessment of objective cognitive functioning requires a thorough test battery, typically including performance-based measures of neuropsychological functioning—particularly in the domains of attention, executive functioning, and memory—, caregiver or collateral ratings of symptoms and functioning, and behavioral observations (Lezak et al., 2012). Subjective patient report of symptoms and functioning are useful to contextualize the objective data and collateral ratings, particularly since both amplification of deficits and lack of awareness of deficits have been observed after TBI (Flashman and McAllister, 2002; Jamora et al., 2012).
Patterns of post-injury recovery of cognitive symptoms have been shown to differ with injury severity, mechanism, and location, with the fastest rate of recovery for both objective cognitive impairments and subjective report of symptoms generally observed in the first 6 months post-injury (Cristofori and Levin, 2015; Model Systems Knowledge Translation Center, 2019). Cognitive symptoms typically resolve rapidly after mild TBI but may be long-lasting—even lifelong—after severe TBI (Schretlen and Shapiro, 2003; Iaccarino et al., 2015).
Subjective experience of cognitive difficulties post-TBI can be evaluated via interview, although formal assessments of patient-reported outcomes (PROs) are useful for settings where standardization is needed (Reeve et al., 2013). Standardization allows for repeated assessment of an individual, which is valuable for monitoring progress in rehabilitation or fluctuation of symptoms over time (Snyder et al., 2012). Standardized PROs are also vital for comparing groups as part of controlled treatment trials (Brundage et al., 2013). Condition-specific PROs can be particularly useful in these settings as they provide a shared context for specificity of the item content and also a relevant clinical comparison sample for score interpretation (Schifferdecker et al., 2018).
Several initiatives have been undertaken in the United States over the past two decades to develop standardized PRO measures for use in healthcare settings. This includes most notably the Patient Reported Outcomes Measurement Information System (PROMIS®) initiative funded by the National Institutes of Health Common Fund and the Quality of Life in Neurological Disorders (Neuro-QoL™) measurement system funded by the National Institute of Neurologic Disorders and Stroke (Fries et al., 2005; Cella et al., 2006, 2007, 2011; National Institutes of Health, 2015; Northwestern University, 2021a). PROMIS and Neuro-QoL measures and their supporting documentation are freely available on the HealthMeasures.net website (Northwestern University, 2021a).
The Traumatic Brain Injury Quality of Life measurement system (TBI-QOL) (Tulsky et al., 2016a; Tulsky and Kisala, 2019) is a TBI-specific measurement system targeted to domains and issues relevant to individuals with TBI. It is composed of item banks designed specifically for individuals with TBI (e.g., Headache Pain, Independence, and Grief/Loss) as well as generic item banks that are relevant to most individuals (e.g., Depression and Anxiety). For the generic item banks, the TBI-QOL measures are optimized versions of the PROMIS and Neuro-QoL measurement systems. The TBI-QOL was developed as part of TBI Model System Collaborative research project beginning in 2007 with support from the National Institute on Disability, Independent Living, and Rehabilitation Research (formerly the National Institute on Disability and Rehabilitation Research) (Tulsky et al., 2016a). The foundation of the content domains and item content included in the TBI-QOL measures are grounded in qualitative work with TBI patients, clinicians, and caregivers (Carlozzi et al., 2011). The TBI-QOL includes 20 item banks across physical, emotional, social, and cognitive domains (Tulsky and Kisala, 2019). All TBI-QOL item banks have been calibrated using item response theory (IRT) and are available as computer adaptive tests (CATs) that can rapidly evaluate and monitor individuals following TBI (Tulsky et al., 2016b,2019; Capo-Lugo et al., 2019; Carlozzi et al., 2019, 2020; Cohen et al., 2019; Kisala et al., 2019, 2020; Heinemann et al., 2020; Sherer et al., 2020). Composite scores for each of the four domains and for global QOL are also available (Tyner et al., 2020). The TBI-QOL measures can be accessed and administered as fixed-length forms or CATs via REDCap (Harris et al., 2009, 2019), or through the NIH Toolbox and PROMIS iPad Apps (Northwestern University, 2021b). Furthermore, the item banks and short forms can be obtained by contacting the authors at VEJJLVFPTEB1ZGVsLmVkdQ==. Details on the TBI-QOL measures and composite scores are presented in Table 1.
Table 1. Traumatic Brain Injury Quality of Life (TBI-QOL) item bank content, item counts, and scoring information.
The TBI-QOL cognition item banks are of particular interest for neuropsychological rehabilitation, given the need for standardized, reliable, and validated outcomes measures that can evaluate subjective symptoms during and following rehabilitation. The TBI-QOL cognition item banks are therefore the focus of this psychometric study. These banks include newly written items as well as items that were drawn from the Neuro-QoL v1.0 cognition item banks. A large effort of qualitative work in TBI was undertaken to develop new TBI-specific items to build these new measures (Carlozzi et al., 2019). With the newly written items, additional focus was placed on the domains of problem solving, cognitive flexibility, learning/memory, attention/concentration, and processing speed (Carlozzi et al., 2019). The structure of the item division into two banks (Executive Function and Cognition–General Concerns) was maintained to preserve cross-condition comparability with the Neuro-QoL v1.0 item banks of the same name. All items were calibrated using IRT based on data gathered from individuals with complicated mild, moderate, or severe TBI (Tulsky et al., 2016a; Tulsky and Kisala, 2019). Thus, both the TBI-QOL Cognition–General Concerns and TBI-QOL Executive Function item banks are optimized for people with TBI. The scores on these two TBI-QOL cognition measures have been transformed and equated to the Neuro-QoL item banks so that both of these item banks are directly comparable to a score on the parallel Neuro-QoL v1.0 item bank. The strong test-retest reliability of the TBI-QOL cognition measures has been established previously [i.e., Executive Function: r = 0.86 (95% CI = 0.79–0.91); Cognition–General Concerns: r = 0.88 (95% CI = 0.81–0.92)] (Carlozzi et al., 2019).
The responsiveness of a PRO measure is an aspect of construct validity that estimates the ability of a measure to capture meaningful change over time (Liang, 2000; Terwee et al., 2003; Revicki et al., 2006; King, 2011). Minimal important differences (MIDs) can be computed to facilitate interpretation of changes in PRO scores over time (Revicki et al., 2008). Neither responsiveness nor MID are fixed characteristics of PRO measures, as they each will vary in different groups and settings. For a PRO to be recommended for a specific use, the responsiveness of the measure to detect meaningful change must be demonstrated in the target population (Roach, 2006). Thus, use of multiple methods of evaluation is recommended to lend support to responsiveness of a measure (Revicki et al., 2008). Preliminary evidence of responsiveness of the TBI-QOL measures have been shown in a community-dwelling sample comprised of individuals who were an average of 5 years post-injury (Poritz et al., 2020). This study showed general stability of TBI-QOL scores over a 6-month study period, with a small but significant improvement in TBI-QOL Executive Functioning item bank scores (i.e., average of 2.7 T-score points) observed in individuals who reported improved participation across the 6-month study period on the Participation Assessment with Recombined Tools–Objective (PART-O) (Poritz et al., 2020).
As part of the process of marshaling evidence for the construct validity and utility of the TBI-QOL, the present study was designed to evaluate the responsiveness of the TBI-QOL cognition measures in a sample of individuals with recent TBI (i.e., <6 months post-injury). Patient report of cognitive functioning and symptoms from this sample provides a unique opportunity for evaluating measure responsiveness, as improvement in self-reported cognition functioning would be expected for many individuals due to the typical course of recovery after TBI. Additionally, this will be the first study to evaluate the responsiveness of the TBI-QOL Cognitive Health Composite score, which is one of five recently developed summary metrics for the TBI-QOL measures (Tyner et al., 2020). Although there is some disagreement about the best metrics to use for evaluating the responsiveness of outcomes measures (Angst, 2011), this study sought to examine responsiveness following the recommendations of the COSMIN guidelines (Terwee et al., 2003; Mokkink et al., 2010), which recommend testing differences in change scores between groups (Mokkink et al., 2010), in addition to evaluating both distribution- and anchor-based metrics of MID (Revicki et al., 2008; King, 2011).
Materials and Methods
Participants
Participants were 134 adults with a history of recent TBI recruited from three clinical sites in the United States as part of a larger multi-year study to evaluate longitudinal trajectories of recovery after TBI. All three sites were part of the TBI Model Systems program, which is a network of institutions that are funded by the United States National Institute on Disability, Independent Living, and Rehabilitation Research as leading centers for clinical care and research on TBI (Model Systems Knowledge Translation Center, 2021). Criteria for enrollment in the TBI Model Systems are described elsewhere and include meeting a standard definition of a moderate or severe TBI with objective medical indicators and being age 16 or older at time of injury (National Data and Statistical Center of the Traumatic Brain Injury Model System, 2007; Corrigan et al., 2012). This study also included individuals with a history of complicated mild TBI, as defined by the presence of abnormal structural neuroimaging in the context of an otherwise mild TBI. Only participants injured less than 6 months prior to enrollment were included in the present study. Only participants with capacity to provide informed consent, as determined by the referring physician or psychologist, were enrolled. Additional inclusion criteria for this study were age 18 or older at time of consent, ability to read, speak and understand English, and ability to understand and respond (by voice) to PRO items. Medical records were reviewed to confirm diagnosis of TBI. Injury severity was categorized as complicated mild, moderate, or severe based on available medical record information (i.e., Glasgow Coma Scale score in emergency department, duration of loss of consciousness, duration of post-traumatic amnesia, and neuroimaging findings) using standard classification schema (Williams et al., 1990; O’Neil et al., 2013). The institutional review board or ethics committee at each site reviewed and approved this project.
Measures and Procedures
Data collection was conducted either as part of an in-person or telephone interview where participants received select measures from the TBI-QOL (i.e., Depression, Anxiety, Resilience, Anger, Emotional and Behavioral Dyscontrol, Headache Pain, Fatigue, Executive Functioning, and Cognition–General Concerns) as CATs with an eight-item minimum and 12-item maximum. These measures yield T-scores with a mean of 50 and standard deviation (SD) of 10. Since the TBI-QOL Cognition–General Concerns and Executive Function banks have been linked to the Neuro-QoL v1.0 metric, the reference sample for these item banks is a combined general population and clinical neurological sample (Gershon et al., 2012). Additional details on CAT administration, calibration, and normative references for the TBI-QOL measures is provided elsewhere and will not be repeated here (Tulsky and Kisala, 2019).
There are 37 Executive Function and 39 Cognition–General Concerns items calibrated in these banks; the item stems (i.e., the phrasing of each question), item contexts (e.g., “In the past 7 days…”), and response sets (e.g., Never/Rarely/Sometimes/Often/Always) for these two banks have been published previously and will not be repeated here (Carlozzi et al., 2019). The TBI-QOL Cognitive Health Composite score for each assessment was calculated using the lookup tables provided by the developers (Tyner et al., 2020), whereby the T-scores from the Executive Function and Cognition–General Concerns measures are summed and converted to an index score. The composite scores are in standard score units, with a mean of 100 and a SD of 15, and are based on a cohort of persons with TBI (Tyner et al., 2020). Other traditional assessments of resilience, depression, anxiety, participation, life satisfaction, general health, anger, and fatigue were also administered as part of a larger data collection effort and were not included in the present analyses.
Participants completed the TBI-QOL measures in interview format at the initial assessment (i.e., at time of enrollment) and then a second time 1 year later. At the 1-year follow-up assessment, participants also completed a series of anchor items designed to capture participant perception of any change in symptoms or functioning since the initial assessment (Coon and Cook, 2018). This approach for documenting responsiveness of PRO measures is widely used (Revicki et al., 2008; Ousmen et al., 2018), and provides the benefit of centering the patients’ perspective of their functioning. Anchor ratings were made using a 7-point Likert scale coded as: 1 = Much Worse, 2 = Worse, 3 = A Little Worse, 4 = About the Same, 5 = A Little Better, 6 = Better, 7 = Much Better. For the purpose of this study, only the three TBI-QOL cognition scores and 12 anchor items on the topics of health, well-being, and several areas of cognitive functioning and cognition-related symptoms were analyzed. The wide array of anchor items examined in this study were chosen either because they directly reflect the content of the cognition item banks (i.e., self-perception of various cognitive abilities) or because they cover topics known to be associated with cognition (i.e., overall health, emotional well-being, physical health, and fatigue), as has been recommended (Revicki et al., 2008). This broad approach was selected to be able to detect which, if any, symptom areas the TBI-QOL measures are most responsive to.
Standardization and Quality Control
Data was collected by trained research assistants. Training of all data collectors at the three sites was centralized through a single project investigator (PK) to ensure consistency and standardization across data collectors and sites. Each data collector attended an interactive, web-based training session, passed a certification quiz, and completed multiple practice assessments prior to beginning live data collection. Data collectors met biweekly to review recruitment and accrual progress and discuss any actual or anticipated changes or challenges. Data collectors were trained to not provide assistance to participants in selecting their responses. Instead, interviewers simply read the questions aloud and directed the participant to the appropriate response card to select the response that they felt most closely fit their experience. All data were entered directly by the interviewer into the Assessment CenterSM online platform (Gershon et al., 2010). The use of this platform and the interview format helped to avoid within-interview missing data (i.e., inadvertent skipping of questions by participants). Data were downloaded and reviewed for quality and completeness at least biweekly. We flagged and reviewed any cases that appeared to fall short of quality standards. There was a case of improbable time stamps for data entry that was traced to a single data collector. After extensive investigation and out of an abundance of caution, six baseline and seven 1-year follow-up observations collected by that single data collector were removed from the dataset.
Data Analyses
Standard error of measurement [SEM; computed as SD × √(1 − r)] (Wyrwich et al., 1999; King, 2011) and minimum detectable change (MDC95%; computed as 1.96 × √2 × SEM) (Beaton et al., 2001; de Vet et al., 2006; King, 2011) were computed as distribution-based methods for evaluating the MID of the TBI-QOL Cognition–General Concerns, Executive Function, and Cognitive Health Composite scores. The standard response mean (SRM) (Kazis et al., 1989) was computed as a measure of effect size for the mean difference in these three scores from the initial and 1-year follow-up assessments (computed as Mdifference/SDdifference). SRM values ≥0.3 were considered indicative of a moderate effect size corresponding to meaningful change (Cohen, 1988). Paired sample t-tests were used to evaluate the statistical difference between the TBI-QOL scores at the two time points.
Responsiveness was further evaluated by examining associations between each of the 12 anchor item ratings and TBI-QOL change scores (i.e., difference between initial and 1-year follow-up assessment scores on the Cognition–General Concerns, Executive Functioning, and Cognitive Health Composite) using Pearson correlations. Correlation coefficient values ≥0.30 indicated a meaningful, moderate relationship (Coon and Cook, 2018; Ousmen et al., 2018). The Benjamini–Hochberg procedure was used to correct statistical significance values to account for multiple comparisons (Benjamini and Hochberg, 1995).
Associations of the TBI-QOL change scores with clinical indicators of injury severity at the initial assessment were also evaluated as markers of responsiveness. Group differences in change scores by TBI injury severity (i.e., complicated mild, moderate, and severe) were compared using one-way ANOVA. Associations of TBI-QOL change scores with other markers of injury severity and recovery pace (i.e., time since injury; time to follow commands; acute care length of stay (LOS); inpatient rehabilitation LOS; and living situation at time of initial evaluation) were examined using Pearson correlations, independent samples t-tests, and exploratory stepwise multiple regression analyses. Exploratory analysis of patterns of variation in TBI-QOL change scores with demographic variables were also evaluated using Pearson correlations (age) and one-way ANOVAs (gender, race, ethnicity, and recruitment site) to understand any potential covariates. All analyses were conducted using SPSS Version 26 (IBM Corp, 2019).
Results
Data from 128 participants with a medical record confirmed diagnosis of complicated mild (40%), moderate (21%), or severe (31%) TBI with a baseline assessment within 6 months of injury were analyzed. Injury severity was unknown or missing in 8% of cases. The majority of participants in the sample were male (60%), white (69%), and non-Hispanic (73%). The participants ranged in age from 18 to 92 years of age (mean 51.4) and were first evaluated from 0 to 5 months post-injury (mean 2.0 months). Additional demographic and injury severity indicators are shown in Table 2. Eighty participants completed the 1-year follow-up assessment.
Table 2. Participant demographic and injury characteristics.
Mean TBI-QOL cognition measure scores and SEM values for each time point are presented in Table 3. Change scores showed improvements of 4.0 and 3.9 T-score points on average after 1 year on the Cognition–General Concerns and Executive Function measures, respectively. Mean improvement of 7.3 standard score points was observed on the Cognitive Health Composite. MDC95% values were 8.8 and 8.2 T-score points on the Cognition–General Concerns and Executive Function measures. SRM values ranged from 0.47 to 0.51. Paired samples t-tests indicated significant improvement in self-reported cognition after 1 year on both of the TBI-QOL cognition item banks and the Cognitive Health Composite (p ≤ 0.001).
Table 3. TBI-QOL cognition scores, distribution-based change metrics, and paired sample t-test results.
The majority of participants reported improvement over the preceding year on the 12 anchor items examined. Table 4 shows the mean change in Cognitive Health Composite scores (in standard score units) for individuals who reported Worse, Same, or Better functioning. Pearson correlation results, shown in Table 5, indicate that three of the 12 anchor items were related with change on the TBI-QOL Executive Function measure and the Cognitive Health Composite, but not the Cognition–General Concerns measure. Using the standard of r ≥ 0.3 to indicate a meaningful, moderate correlation, three anchor items were found to be associated with meaningful self-reported improvement in cognition after 1 year: ability to pay attention; ability to multitask; and ability to remember. Considering mean change scores for these three anchor items, the observed mean Cognitive Health Composite score changes (as shown in Table 4) ranged from 10.0 to 10.8 standard score points for those reporting improvements in self-reported cognition over the preceding year (i.e., about two-thirds of a SD). For those reporting no change, mean score changes ranged from −1.7 to 2.9, and for those reporting worsening of cognitive function, mean score changes ranged from −20.0 to 4.5 points, although these latter estimates are based on few individuals and are not considered reliable.
Table 4. TBI-QOL Cognitive Health Composite score mean change by anchor item response groupings.
Table 5. Pearson correlations of anchor item responses with TBI-QOL change scores after 1 year.
One-way ANOVAs of TBI-QOL change scores by injury severity classification revealed significant (p ≤ 0.05) differences in mean change scores by TBI severity group for the Cognition–General Concerns and Cognitive Health Composite (see Table 6), although the effect sizes were small (i.e., 0.23 for both). A box and whisker plot comparing the severity groups is presented in Figure 1. Considering mean change score by injury severity, the complicated mild injury group improved by 0.5–0.7 of a SD over the 1-year study period, the moderate injury group improved by 0.4 of a SD, and the severe injury group improved by 0.1–0.2 of a SD. Tukey post hoc contrasts revealed significant differences between the complicated mild and severe groups for Cognition–General Concerns (p = 0.025) and the Cognitive Health Composite (p = 0.034). To more fully parse the variation in change score by injury severity, the mean score change on the Cognitive Health Composite was compared for individuals of different injury severities by response to the three salient anchor items: attention, multitasking, and memory. Supplementary Table 1 shows that the magnitude of the composite score change (in standard score units) was the largest for those reporting better functioning in the complicated mild group (12.6–15.8), over the moderate (10.7–11.7) and severe (3.8–5.3) groups. Likewise, the mean change score for those reporting same/worse functioning was most negative for the severe group (−2.8 to −1.5) over the moderate (−1.8 to −0.8) group; those in the complicated mild group endorsing same or worse functioning actually showed small improvements on average (1.0–3.6).
Table 6. TBI-QOL cognition change scores by injury severity.
Figure 1. Box and whisker plot of score changes by TBI severity group. Cognition-General Concerns and Executive Functioning are in T-score units (M = 50, SD = 10), while Cognitive Health Composite is in standard score units (M = 100, SD = 15). Box and whisker plot displays error bars to indicate the score range (min to max), with the rectangular box showing the interquartile range, the central line indicating the median value, and the X marking the mean value.
Evaluation of other markers of injury severity and recovery revealed weak to negligible associations of these clinical variables with TBI-QOL change scores (see Table 7). There were no significant differences in mean change score by different living situations at the initial evaluation [with participants grouped as either (1) living in initial inpatient rehabilitation, other rehabilitation, or long-term care vs. (2) private residence]. To evaluate for potential complex effects of these clinical indicators on patterns of change, exploratory stepwise multiple regression analyses were conducted for the three TBI-QOL change scores, although no models were statistically significant. There were no significant associations or patterns of mean difference observed with any other demographic variables (i.e., correlations between change scores with age and one-way ANOVAs of gender, race, ethnicity, and recruitment site all ps > 0.2).
Table 7. Associations of TBI-QOL cognition change scores with clinical indicators at initial assessment.
Discussion
This study provides evidence to support the responsiveness of the TBI-QOL cognition item banks and Cognitive Health Composite score for use in individuals within the first 18 months of recovery post-TBI. Improvements in self-reported cognition were endorsed for the majority of participants in this sample, on both the TBI-QOL cognition measures and using a set of anchor items. Meaningful associations with TBI-QOL Executive Functioning and Cognitive Health Composite score change were observed with anchor ratings of attention, multitasking, and memory. Individuals who endorsed improvements in these domains on anchor ratings improved by about two-thirds of a SD on the TBI-QOL Cognitive Health Composite score. These findings are generally consistent with previous research showing marked improvements in the domains of executive functioning and memory, particularly in the early recovery period, following TBI (Christensen et al., 2008). There were no meaningful correlations between the Cognition–General Concerns change scores and the anchor items, which is unexpected given that the items on this measure were designed to be sensitive to a wide variety of self-reported cognitive symptoms. Although still below the threshold of ≥0.30 to indicate meaningful, moderate relationships, the correlations were statistically significant for all three TBI-QOL measures with fatigue. TBI-QOL score improvements were not consistently associated with self-reported emotional well-being or physical health. The weak association between the TBI-QOL change scores with the anchor ratings of general (non-specific) cognitive functioning, concentration, problem solving, learning, and communication may be due to restriction of range of responses to the anchors. For many of the cognition-related anchor items, no participants endorsed worse functioning. Worse functioning was only reported by four participants for cognitive functioning, one for multitasking, and six for memory. With a larger sample we anticipate that the associations between anchor items and change scores would be more robust.
The MDC values reported in this study are somewhat conservative, as they reflect the amount of change in score where there is 95% certainty statistically that someone would have experienced change in their subjective cognitive functioning. Some authors have suggested that using this 95% certainty criterion is overly conservative in clinical samples, as some individuals who have experienced meaningful change will not be detected (Calfee and Ryan, 2019; Cohen et al., 2021). Using a less conservative method of computation, for example the MDC 90% or using a one-tailed approach, may provide a more accurate estimate of the true MDC (Cohen et al., 2021). For example, in this study the MDC95% values reported (see Table 3) were only met by 15–16% of the sample. If a standard of MDC90% were applied, 7.4 and 6.9 for Cognition–General Concerns and Executive Function, respectively, then 20% of the sample would exceed these thresholds of statistically significant change in self-reported cognitive QOL. These low numbers could be an underestimate of the true number of cases who experienced improved cognitive QOL over this study period. Thus, the MDC values reported in this study should be interpreted with caution in individual cases, given that they are likely overly conservative and some individuals who have experienced important change in cognitive QOL will be missed.
There were small but significant differences in TBI-QOL score changes by injury severity for the Cognition–General Concerns and Cognitive Health Composite, with the greatest score improvement observed in the complicated mild injury group, followed by moderate, and with those in the severe injury group showing the smallest average change scores. This pattern is consistent with previous research showing a greater likelihood for those with mild TBI to return to at or near their pre-injury cognitive status within 6 months to a year, compared to those with moderate or severe injuries where persistent cognitive symptoms are much more common, and recovery takes place over years (Schretlen and Shapiro, 2003; Iaccarino et al., 2015). Furthermore, this finding suggests that without any formal cognitive intervention provided, 1 year may not be long enough to observe changes in self-reported cognitive functioning for newly injured individuals with severe TBI, whereas individuals with complicated mild and moderate injuries are more likely to endorse improvements in cognitive functioning in the first year post-injury in the absence of intervention. This is consistent with previous research showing differences in recovery time course that vary with injury severity, with most individuals after mild TBI showing the bulk of their cognitive symptom improvements in the first few months post-injury and individuals with severe TBI often showing signs of continued improvement as far out as 2 years post-injury (Schretlen and Shapiro, 2003; Iaccarino et al., 2015; Wilkins et al., 2019).
Limitations
One limitation of this study is that, as an observational study, the expectation for measurable improvement in self-report of cognitive symptoms during the study period may not be as certain as might have been observed had this been a treatment trial. This is demonstrated by the large, and likely overly conservative, MDC values reported. Future research in the context of an intervention study could further support the responsiveness of these measures. Low sample sizes observed in the anchor rating categories, particularly in the worse categories, is a related potential limitation. Given that some amount of cognitive recovery is expected for the majority of individuals following TBI (Schretlen and Shapiro, 2003), we would not have expected many participants to have reported worsening of symptoms at follow-up. Nevertheless, it has been suggested that the presence of low sample sizes in certain anchor categories may be associated with reduced reliability of MID estimates (Ousmen et al., 2018). This study is also limited in that self-report was used for both assessment of changes in cognitive functioning over time and cognitive-related QOL ratings. It is not surprising that people who rate their cognition as worse or unimproved would also rate their cognition-related QOL as poor, and vice versa. Future research should determine the association between objective cognitive test performance and cognitive-related QOL. Furthermore, impairments in self-awareness are common during the early recovery period, particularly in persons with more severe injuries (Flashman and McAllister, 2002). Impaired awareness should be considered in future research investigating subjective reports of cognitive functioning and QOL. Moreover, the high rate of attrition observed at the 1-year follow-up further limits the current results. Therefore, replication of these findings, particularly in a sample as part of a controlled treatment trial, is warranted.
Conclusion
The findings support the responsiveness of the TBI-QOL Cognition–General Concerns, Executive Function, and Cognitive Health Composite score for use in newly injured individuals. The results presented here provide a variety of distribution- and anchor-based metrics of MID that can be used for score interpretation by clinicians and researchers seeking PRO measures of self-reported cognition after TBI.
Data Availability Statement
The datasets presented in this article are not readily available because a data use agreement must be signed prior to release. Requests to access the datasets should be directed to DT, (ZHR1bHNreUB1ZGVsLmVkdQ==).
Ethics Statement
The studies involving human participants were reviewed and approved by the Institutional Review Boards at NYU Langone Medical Center, Kessler Foundation, Baylor College of Medicine, and the University of Delaware. The patients/participants provided their written informed consent to participate in this study.
Author Contributions
PK, MS, NC, AS, TB, and DT participated in data collection. AB and PK managed the dataset. CT performed the statistical analysis and wrote the first draft of the manuscript. DT, PK, AB, and AS wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.
Funding
This work was supported by grant numbers 90DP0028 (formerly H133A120020), 90DP0032 (formerly H133A120030), 90DP0047 (formerly H133A120100) from The National Institute on Disability, Independent Living, and Rehabilitation Research.
Conflict of Interest
All TBI-QOL items, parameters, and data are ©2016 David Tulsky. All rights reserved. All TBI-QOL items originally from Neuro-QoL are ©2008–2013 David Cella on behalf of the National Institute for Neurological Disorders and Stroke. All items are freely available to the public upon request (contact VEJJLVFPTEB1ZGVsLmVkdQ==). The contents represent original work and have not been published elsewhere.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum.2022.763311/full#supplementary-material
Abbreviations
CAT, computer adaptive test; ES, effect size; IRT, item response theory; LOS, length of stay; MDC, minimum detectable change; MID, minimal important difference; Neuro-QoL™, Quality of Life in Neurological Disorders; PART-O, Participation Assessment with Recombined Tools–Objective; PRO, patient-reported outcome; PROMIS®, Patient Reported Outcomes Measurement Information System; QOL, quality of life; SEM, standard error of measurement; SRM, standard response mean; TBI, traumatic brain injury.
References
Angst, F. (2011). The new COSMIN guidelines confront traditional concepts of responsiveness. BMC Med. Res. Methodol. 11:152. doi: 10.1186/1471-2288-11-152
Beaton, D. E., Bombardier, C., Katz, J. N., Wright, J. G., Wells, G., Boers, M., et al. (2001). Looking for important change/differences in studies of responsiveness. OMERACT MCID Working Group. Outcome Measures in Rheumatology. Minimal Clinically Important Difference. J. Rheumatol. 28, 400–405.
Benjamini, Y., and Hochberg, Y. (1995). Controlling the False Discovery Rate: a Practical and Powerful Approach to Multiple Testing. J. Roy. Stat. Soc. B. Met. 57, 289–300. doi: 10.1111/j.2517-6161.1995.tb02031.x
Brundage, M., Blazeby, J., Revicki, D., Bass, B., de Vet, H., Duffy, H., et al. (2013). Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards. Qual. Life Res. 22, 1161–1175. doi: 10.1007/s11136-012-0252-1
Calfee, R. P., and Ryan, P. (2019). Calfee replies to Dr Terwee. J. Hand. Surg. Am. 44, e3. doi: 10.1016/j.jhsa.2019.10.027
Capo-Lugo, C. E., Kisala, P. A., Boulton, A. J., Choi, S. W., Heinemann, A. W., and Tulsky, D. S. (2019). Measuring Self-Reported Physical Function in Individuals With TBI: development of the TBI-QOL Mobility and Upper Extremity Item Banks and Short Forms. J. Head Trauma Rehabil. 34, 340–352. doi: 10.1097/HTR.0000000000000511
Carlozzi, N. E., Kisala, P. A., Boulton, A. J., Roth, E., Kratz, A. L., Sherer, M., et al. (2020). Measuring Pain in TBI: development of the TBI-QOL Pain Interference Item Bank and Short Form. Arch. Phys. Med. Rehabil. 101, 11–19. doi: 10.1016/j.apmr.2019.07.019
Carlozzi, N. E., Tulsky, D. S., and Kisala, P. A. (2011). Traumatic brain injury patient-reported outcome measure: identification of health-related quality-of-life issues relevant to individuals with traumatic brain injury. Arch. Phys. Med. Rehabil. 92(10 Suppl.), S52–S60. doi: 10.1016/j.apmr.2010.12.046
Carlozzi, N. E., Tyner, C. E., Kisala, P. A., Boulton, A. J., Sherer, M., Chiaravalloti, N., et al. (2019). Measuring Self-Reported Cognitive Function Following TBI: development of the TBI-QOL Executive Function and Cognition-General Concerns Item Banks. J. Head Trauma Rehabil. 34, 308–325. doi: 10.1097/HTR.0000000000000520
Cella, D., Nowinski, C., Peterman, A., Victorson, D., Miller, D., Lai, J. S., et al. (2011). The neurology quality-of-life measurement initiative. Arch. Phys. Med. Rehabil. 92(10 Suppl.), S28–S36.
Cella, D., Victorson, D., Nowinski, C., Peterman, A., and Miller, D. M. (2006). The Neuro-QOL project: using Multiple Methods to Develop a HRQOL Measurement Platform to be Used in Clinical Research Across Neurological Conditions. Qual Life Res A-14:1353.
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., et al. (2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med. Care. 45(5 Suppl. 1), S3–S11. doi: 10.1097/01.mlr.0000258615.42478.55
Christensen, B. K., Colella, B., Inness, E., Hebert, D., Monette, G., Bayley, M., et al. (2008). Recovery of cognitive function after traumatic brain injury: a multilevel modeling analysis of Canadian outcomes. Arch. Phys. Med. Rehabil. 89(12 Suppl.), S3–S15. doi: 10.1016/j.apmr.2008.10.002
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.
Cohen, M. L., Kisala, P. A., Boulton, A. J., Carlozzi, N. E., Cook, C. V., and Tulsky, D. S. (2019). Development and Psychometric Characteristics of the TBI-QOL Communication Item Bank. J. Head Trauma Rehabil. 34, 326–339. doi: 10.1097/HTR.0000000000000528
Cohen, M. L., Lanzi, A. M., and Boulton, A. J. (2021). Clinical Use of PROMIS, Neuro-QoL, TBI-QoL, and Other Patient-Reported Outcome Measures for Individual Adult Clients with Cognitive and Language Disorders. Semin. Speech Lang. 42, 192–210. doi: 10.1055/s-0041-1731365
Coon, C. D., and Cook, K. F. (2018). Moving from significance to real-world meaning: methods for interpreting change in clinical outcome assessment scores. Qual. Life Res. 27, 33–40. doi: 10.1007/s11136-017-1616-3
Corrigan, J. D., Cuthbert, J. P., Whiteneck, G. G., Dijkers, M. P., Coronado, V., Heinemann, A. W., et al. (2012). Representativeness of the Traumatic Brain Injury Model Systems National Database. J Head Trauma Rehabil. 27, 391–403. doi: 10.1097/htr.0b013e3182238cdd
Cristofori, I., and Levin, H. S. (2015). Traumatic brain injury and cognition. Handb. Clin. Neurol. 128, 579–611.
de Vet, H. C., Terwee, C. B., Ostelo, R. W., Beckerman, H., Knol, D. L., and Bouter, L. M. (2006). Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change. Health Qual Life Outcomes. 4:54. doi: 10.1186/1477-7525-4-54
Flashman, L. A., and McAllister, T. W. (2002). Lack of awareness and its impact in traumatic brain injury. NeuroRehabilitation. 17, 285–296.
Fries, J. F., Bruce, B., and Cella, D. (2005). The promise of PROMIS: using item response theory to improve assessment of patient-reported outcomes. Clin. Exp. Rheumatol. 23(5 Suppl. 39), S53–S57.
Gershon, R., Rothrock, N. E., Hanrahan, R. T., Jansky, L. J., Harniss, M., and Riley, W. (2010). The development of a clinical outcomes survey research application: assessment Center. Qual Life Res. 19, 677–685. doi: 10.1007/s11136-010-9634-4
Gershon, R. C., Lai, J. S., Bode, R., Choi, S., Moy, C., Bleck, T., et al. (2012). Neuro-QOL: quality of life item banks for adults with neurological disorders: item development and calibrations based upon clinical and general population testing. Qual Life Res. 21, 475–486. doi: 10.1007/s11136-011-9958-8
Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O’Neal, L., et al. (2019). The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 95:103208. doi: 10.1016/j.jbi.2019.103208
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., and Conde, J. G. (2009). Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support. J. Biomed. Inform. 42, 377–381. doi: 10.1016/j.jbi.2008.08.010
Heinemann, A. W., Kisala, P. A., Boulton, A. J., Sherer, M., Sander, A. M., Chiaravalloti, N., et al. (2020). Development and Calibration of the TBI-QOL Ability to Participate in Social Roles and Activities and TBI-QOL Satisfaction With Social Roles and Activities Item Banks and Short Forms. Arch. Phys. Med. Rehabil. 101, 20–32. doi: 10.1016/j.apmr.2019.07.015
Iaccarino, M. A., Bhatnagar, S., and Zafonte, R. (2015). “Chapter 26 - Rehabilitation after traumatic brain injury,” in Handb Clin Neurol, eds J. Grafman and A. M. Salazar (Amsterdam: Elsevier), 411–422.
Jamora, C. W., Young, A., and Ruff, R. M. (2012). Comparison of subjective cognitive complaints with neuropsychological tests in individuals with mild vs more severe traumatic brain injuries. Brain Inj. 26, 36–47. doi: 10.3109/02699052.2011.635352
Kazis, L. E., Anderson, J. J., and Meenan, R. F. (1989). Effect sizes for interpreting changes in health status. Med. Care 27(3 Suppl.), S178–S189. doi: 10.1097/00005650-198903001-00015
King, M. T. A. (2011). point of minimal important difference (MID): a critique of terminology and methods. Expert Rev Pharmacoecon. Outcomes Res. 11, 171–184. doi: 10.1586/erp.11.9
Kisala, P. A., Bushnik, T., Boulton, A. J., Hanks, R. A., Kolakowsky-Hayner, S. A., and Tulsky, D. S. (2019). Measuring Fatigue in TBI: development of the TBI-QOL Fatigue Item Bank and Short Form. J. Head Trauma Rehabil. 34, 289–297. doi: 10.1097/HTR.0000000000000530
Kisala, P. A., Tulsky, D. S., Boulton, A. J., Heinemann, A. W., Victorson, D., Sherer, M., et al. (2020). Development and Psychometric Characteristics of the TBI-QOL Independence Item Bank and Short Form and the TBI-QOL Asking for Help Scale. Arch. Phys. Med. Rehabil. 101, 33–42. doi: 10.1016/j.apmr.2019.08.469
Lezak, M. D., Howieson, D. B., Bigler, E. D., and Tranel, D. (2012). Neuropsychological Assessment. Fifth ed. Oxford: Oxford University Press.
Liang, M. H. (2000). Longitudinal construct validity: establishment of clinical meaning in patient evaluative instruments. Med. Care 38(9 Suppl.), II84–II90.
Model Systems Knowledge Translation Center. (2019). Understanding TBI: Part 3 - The Recovery Process. Available online at : https://msktc.org/tbi/factsheets/Understanding-TBI/The-Recovery-Process-For-Traumatic-Brain-Injury (accessed June 29, 2021).
Model Systems Knowledge Translation Center. (2021)TBI Model Systems | Model Systems Knowledge Translation Center (MSKTC). Available online at: https://msktc.org/tbi/model-system-centers (accessed November 23, 2021).
Mokkink, L. B., Terwee, C. B., Knol, D. L., Stratford, P. W., Alonso, J., Patrick, D. L., et al. (2010). The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Med. Res. Methodol. 10:22. doi: 10.1186/1471-2288-10-22
National Data and Statistical Center of the Traumatic Brain Injury Model System. (2007). Identification of Subjects for the TBI Model Systems National Database. Available online at: https://www.tbindsc.org/StaticFiles/SOP/1%20Data%20Collection/101a%20-%20Identification%20of%20Subjects.pdf (accessed November 23, 2021).
National Institutes of Health. (2015). PROMIS: Clinical Outcomes Assessment. Available online at: https://commonfund.nih.gov/promis/index (accessed November 23, 2021).
Northwestern University. (2021a). HealthMeasures. Available online at: http://www.healthmeasures.net/ (accessed November 23, 2021).
Northwestern University. (2021b)NIH Toolbox and PROMIS iPad Apps. Available online at: https://www.healthmeasures.net/index.php?option=com_content&view=category&layout=blog&id=132&Itemid=936 (accessed November 23, 2021).
O’Neil, M. E., Carlson, K., Storzbach, D., Brenner, L., Freeman, M., Quinones, A., et al. (2013). Complications of Mild Traumatic Brain Injury in Veterans and Military Personnel: A Systematic Review. Washington DC: Department of Veterans Affairs.
Ousmen, A., Touraine, C., Deliu, N., Cottone, F., Bonnetain, F., Efficace, F., et al. (2018). Distribution- and anchor-based methods to determine the minimally important difference on patient-reported outcome questionnaires in oncology: a structured review. Health Qual. Life Outcomes. 16:228. doi: 10.1186/s12955-018-1055-z
Poritz, J. M. P., Sherer, M., Kisala, P. A., Tulsky, D., Leon-Novelo, L., and Ngan, E. (2020). Responsiveness of the Traumatic Brain Injury-Quality of Life (TBI-QOL) Measurement System. Arch. Phys. Med. Rehabil. 101, 54–61. doi: 10.1016/j.apmr.2017.11.018
Rabinowitz, A. R., and Levin, H. S. (2014). Cognitive sequelae of traumatic brain injury. Psychiatr Clin. North Am. 37, 1–11.
Reeve, B. B., Wyrwich, K. W., Wu, A. W., Velikova, G., Terwee, C. B., Snyder, C. F., et al. (2013). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 22, 1889–1905. doi: 10.1007/s11136-012-0344-y
Revicki, D., Hays, R. D., Cella, D., and Sloan, J. (2008). Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J. Clin. Epidemiol. 61, 102–109. doi: 10.1016/j.jclinepi.2007.03.012
Revicki, D. A., Cella, D., Hays, R. D., Sloan, J. A., Lenderking, W. R., and Aaronson, N. K. (2006). Responsiveness and minimal important differences for patient reported outcomes. Health Qual Life Outcomes. 4:70. doi: 10.1186/1477-7525-4-70
Roach, K. E. (2006). Measurement of Health Outcomes: Reliability, Validity and Responsiveness. Journal of Prosthetics and Orthotics. 18, 8–12.
Schifferdecker, K. E., Yount, S. E., Kaiser, K., Adachi-Mejia, A., Cella, D., Carluzzo, K. L., et al. (2018). A method to create a standardized generic and condition-specific patient-reported outcome measure for patient care and healthcare improvement. Qual Life Res. 27, 367–378. doi: 10.1007/s11136-017-1675-5
Schretlen, D. J., and Shapiro, A. M. (2003). A quantitative review of the effects of traumatic brain injury on cognitive functioning. Int. Rev. Psychiatr. 15, 341–349. doi: 10.1080/09540260310001606728
Sherer, M., Poritz, J. M. P., Tulsky, D., Kisala, P., Leon-Novelo, L., and Ngan, E. (2020). Conceptual Structure of Health-Related Quality of Life for Persons With Traumatic Brain Injury: confirmatory Factor Analysis of the TBI-QOL. Arch. Phys. Med. Rehabil. 101, 62–71. doi: 10.1016/j.apmr.2017.04.016
Snyder, C. F., Aaronson, N. K., Choucair, A. K., Elliott, T. E., Greenhalgh, J., Halyard, M. Y., et al. (2012). Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 21, 1305–1314. doi: 10.1007/s11136-011-0054-x
Terwee, C. B., Dekker, F. W., Wiersinga, W. M., Prummel, M. F., and Bossuyt, P. M. (2003). On assessing responsiveness of health-related quality of life instruments: guidelines for instrument evaluation. Qual Life Res. 12, 349–362. doi: 10.1023/a:1023499322593
Tulsky, D. S., and Kisala, P. A. (2019). An Overview of the Traumatic Brain Injury-Quality of Life (TBI-QOL) Measurement System. J. Head Trauma Rehabil. 34, 281–288. doi: 10.1097/HTR.0000000000000531
Tulsky, D. S., Kisala, P. A., Victorson, D., Carlozzi, N., Bushnik, T., Sherer, M., et al. (2016a). TBI-QOL: development and Calibration of Item Banks to Measure Patient Reported Outcomes Following Traumatic Brain Injury. J. Head Trauma Rehabil. 31, 40–51. doi: 10.1097/HTR.0000000000000131
Tulsky, D. S., Kisala, P. A., Holdnack, J., and Cohen, M. (2016b). Measuring psychological functioning following TBI with the TBI-QOL emotional functioning and adjustment item banks [abstract]. Brain Inj. 30, 670–671.
Tulsky, D. S., Tyner, C. E., Boulton, A. J., Kisala, P. A., Heinemann, A. W., Roth, E. J., et al. (2019). Development of the TBI-QOL Headache Pain Item Bank and Short Form. J. Head Trauma Rehabil. 34, 298–307. doi: 10.1097/HTR.0000000000000532
Tyner, C. E., Boulton, A. J., Sherer, M., Kisala, P. A., Glutting, J. J., and Tulsky, D. S. (2020). Development of Composite Scores for the TBI-QOL. Arch. Phys. Med. Rehabil. 101, 43–53. doi: 10.1016/j.apmr.2018.05.036
Wilkins, T. E., Beers, S. R., Borrasso, A. J., Brooks, J., Mesley, M., Puffer, R., et al. (2019). Favorable Functional Recovery in Severe Traumatic Brain Injury Survivors beyond Six Months. J Neurotrauma 36, 3158–3163. doi: 10.1089/neu.2018.6153
Williams, D. H., Levin, H. S., and Eisenberg, H. M. (1990). Mild head injury classification. Neurosurgery 27, 422–428. doi: 10.1097/00006123-199009000-00014
Keywords: acquired brain injury, rehabilitation, assessment, patient-reported outcomes, psychometrics, cognition
Citation: Tyner CE, Kisala PA, Boulton AJ, Sherer M, Chiaravalloti ND, Sander AM, Bushnik T and Tulsky DS (2022) Responsiveness of the Traumatic Brain Injury Quality of Life Cognition Banks in Recent Brain Injury. Front. Hum. Neurosci. 16:763311. doi: 10.3389/fnhum.2022.763311
Received: 23 August 2021; Accepted: 08 February 2022;
Published: 04 March 2022.
Edited by:
Andrew Bateman, University of Essex, United KingdomReviewed by:
Nicole Sugden, Charles Sturt University, AustraliaJane Powell, University of Surrey, United Kingdom
Copyright © 2022 Tyner, Kisala, Boulton, Sherer, Chiaravalloti, Sander, Bushnik and Tulsky. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Callie E. Tyner, Y3R5bmVyQHVkZWwuZWR1