Cochlear Implant Awareness: Development and Validation of a Patient Reported Outcome Measure

Background: Surgical success of cochlear implantation is usually measured through speech perception and quality of life questionnaires. Although these questionnaires cover a broad spectrum of domains, they do not evaluate the consciousness of wearing a cochlear implant (CI) and how this impacts the daily life of patients. To evaluate this concept we aimed to develop and validate a standardized patient reported outcome measure (PROM) for use in cochlear implant users.

Methods: Development and evaluation of the COchlear iMPlant AwareneSS (COMPASS) questionnaire was realized following the COSMIN guidelines in three phases: (1) item generation, (2) qualitative pilot study to ensure relevance, comprehensiveness, comprehensibility, and face validity, and (3) quantitative survey study for the assessment of reliability (test-retest) with 54 participants.

Results: Nine domains of CI awareness were identified through literature research and interviews with experts and patients. These resulted in the formulation of 18 items which were tested with a pilot study, after which 3 items were deleted. The final 15-item COMPASS questionnaire proved to have good validity and satisfactory reliability. The intraclass correlation coefficient calculated for items with continuous variables ranged from 0.66 to 0.89 with seven out of eight items scoring above the acceptable level of 0.7. The Cohen’s kappa calculated for items with nominal variables ranged from −0.4 to 0.78 with 11 (sub)items out of 15 scoring above fair to good agreement. Measurement error analysis for items with continuous variables showed a mean difference of −2.18 to 0.22. The calculated 95% limits of agreement for these items revealed no statistically significant difference between the two administered questionnaires. For items with nominal variables, the percentages of agreement calculated, ranged between 0 and 95%, and 83.3 and 96.6% for positive and negative agreement, respectively.

Conclusion: The COMPASS questionnaire is a valid and reliable PROM for evaluating the cochlear implant awareness, and it can be easily used in routine clinical practice.

Introduction

Cochlear implants (CI’s) are currently the only effective treatment for auditory rehabilitation for patients with severe-to-profound bilateral sensorineural hearing loss (SNHL) with poor speech perception. Since the introduction of this medical device in the 1970s, great advancements have been made regarding the functionality and hardware design. The internal part of the implant, the receiver/stimulator (R/S) device that resides under the skin behind the pinna of the ear, has undergone technological improvements resulting in thinner implants with smaller footprints (Carlson et al., 2012). Comfort of the external parts of the CI use has increased over the years with more discrete designs and lighter speech processors that allow patients to wear their implant throughout the day. Most importantly, the speech perception results have increased greatly, improving quality of life of patients with hearing loss (Gaylor et al., 2013; Loeffler et al., 2014; McRackan et al., 2018).

Despite the wealth of knowledge and research regarding speech perception results and health-related quality of life of CI recipients, little is known about the CI-experience and -awareness by patients. We define awareness of having a cochlear implant as “the state of mind or situation in which the patient is physically conscious he or she is wearing a cochlear implant and how this consciousness impacts their daily life.” There are patient-reported outcome measures (PROMs) assessing CI use such as the Cochlear Implant Management Skills (CIMS-self) survey and the Nijmegen Cochlear Implantation Questionnaire (NICQ) (Hinderink et al., 2000; Bennett et al., 2017). The CIMS-self focuses on device management exclusively, and the NICQ assesses health-related quality-of-life by how sound and speech perception limits a CI-recipient in their daily life. However, these PROMs do not evaluate the (physical) impact of a CI, thus they may fail to capture cochlear implant awareness topics in daily life that are of importance from patient perspective. To our knowledge, no CI-specific PROM has been developed yet that included patients in item development, following the standards of the Patient Reported Outcomes Measurement Information System (PROMIS) or the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) (McRackan et al., 2018).

Cochlear implant awareness could be important for speech recognition results and quality of life of CI recipients. Studies have shown that wear time of the CI affects speech recognition outcomes in pediatric and adult patients (Gagnon et al., 2020; Holder et al., 2020). In addition, previous research on hearing aids has shown that fit and comfort are the second most important factors contributing to non-use of hearing aids (McCormack and Fortnum, 2013). Specifically, the satisfaction of patients with comfort of use, burden during daily activities, sleep disturbances related to location of the implant in relation to the preferred sleeping position, pain, or other discomfort caused by the implant are all contributing factors to reduced wear time. Moreover, there might be an underestimation of the prevalence of above mentioned problems in CI recipients, especially when a significant increase in hearing and communication is achieved using the CI. The benefits of the CI could suppress the concomitant inconvenience that accompanies wearing the processor.

In order to assess the physical awareness of the cochlear implant, we aimed to develop and validate a patient reported outcome measure (PROM) questionnaire.

Materials and Methods

This development and validation study was conducted between December 2019 and April 2021 at the University Medical Center (UMC) Utrecht, in compliance with the principles of the Declaration of Helsinki. This study was exempt from approval of an ethics committee under Dutch law. Exemption was granted by the local ethical committee (Institutional Review Board of the UMC Utrecht) (METC protocol 19-722/C). A three-stage procedure for development and validation of the patient reported outcome measure (PROM) was conducted, in accordance with the COSMIN guidelines (see Figure 1; Mokkink et al., 2010). Participants were recruited at the time of routine control at the CI center UMC Utrecht, and through an open e-mail invitation to patients registered in the CI database of the UMC Utrecht sent by their attending physician. Written informed consent was obtained from all participants.

FIGURE 1

Figure 1. The three phase procedure for development and validation of the COMPASS questionnaire.

Construction of the Concept

We aimed to develop the COchlear iMPlant AwareneSS (COMPASS) questionnaire to assess the awareness of having a cochlear implant as previously defined. The PROM development group consisted of a otorhinolaryngologist, an epidemiologist and a junior researcher. This questionnaire was designed for adult, Dutch speaking, CI recipients. The instrument was developed to be used as a self-administered evaluation tool, in daily clinical practice, for clinical studies, and for comparison within patients over time (possible evolution of awareness). The questionnaire was designed to detect issues in different categories, specifically concerning the external parts of the CI (speech processor and transmitter) and the internal part (the receiver/stimulator device). In order to assess CI awareness, different domains were identified. It is important to distinguish the situation of awareness and how burdensome the awareness is. Therefore, the questionnaire should consist of multiple choice items as well as scale items to measure the burden. With the results of the questionnaire, health care professionals should be able to identify problems that can be solved by adapting the hardware or by counseling.

Phase 1: Item Generation

Qualitative data were obtained by a literature review, a series of interviews with seven specialists in cochlear implantation care, including an otorhinolarygologist, speech therapists and audiologists, and individual interviews with a sample (n = 7) of CI recipients were conducted, to identify and select relevant aspects of CI awareness. Included patients were adult CI recipients that were using their implant for at least 1 year prior to inclusion in order to have adequate experience with everyday use of their implant to contribute to data collection. The semi-structured interviews of approximately 1 h each were recorded and were conducted by a trained investigator (LM) using an interview guide (see Supplementary Material). The recorded interviews were then transcripted verbatim. Content analysis was performed independently by two researchers (LM and IS), by coding the transcripts and then grouping the codes into thematic categories. Data collection was continued until saturation was reached. The emerging domains as well as the pertinence of the findings were discussed within the research group until consensus was reached. The questionnaire is based on a formative model, the indicators (items) define the value of CI awareness (the construct measured).

Phase 2: Pilot Study (Cognitive Debriefing Test)

A pilot study was conducted to assess the content validity of the questionnaire, the comprehensibility and comprehensiveness. The above mentioned experts in the field of cochlear implantation evaluated the content, wording, format, answer options, and intelligibility. Changes were made appropriately. The evaluated questionnaire was administered to ten adult CI patients that were using their implant for at least 3 months prior to inclusion in order to have adequate experience with everyday use of their implant to contribute to data validation. Participants filled out the questionnaire while “thinking aloud,” followed by a semi-structured interview with open-ended questions (see Supplementary Material) that were audio-recorded. This interview was conducted to capture information on the participant’s understanding of the instructions, the intended meaning and clinical relevance of each item, the response options, patients opinion regarding the questionnaire and missing concepts. The time required to fill out the questionnaire was also recorded. Adjustments were made to the questionnaire based on these interviews.

Phase 3: Reliability Study

A quantitative study was conducted to assess the reliability of the final version of the COMPASS questionnaire. The questionnaire was administered twice to 54 adult CI patients, thereby meeting the COSMIN criteria of participants necessary for quantitative validation (Mokkink et al., 2010). These CI patients were using the CI for at least 3 months prior to inclusion in order to have adequate experience with everyday use of their implant to contribute to data validation. Two weeks after the participants filled out and returned the questionnaire, they were send and filled out the same questionnaire again. The questionnaire was distributed on paper or digitally through Castor EDC (version 1.6, Ciwit B.V., Amsterdam, Netherlands), an electronic data capture platform, depending on the patients’ preferences.

Data Analysis

Data was analyzed using IBM SPSS Statistics for Windows (version 26.0.0.1; IBM Corp., Armonk, NY, United States). Reliability (test-retest) was calculated using the interclass correlation coefficient (ICC) for continuous scores and Cohen’s Kappa with standard error and 95% confidence interval for nominal scores. We used the two-way random effect model with interaction for the absolute agreement between single scores to calculate the ICC with 95% confidence interval. This model was chosen because time is a relevant factor for the test-retest assessment, and because the results will be generalized beyond the study points. Also the participants are assumed to be stable for the construct of interest across the two time points (Qin et al., 2019). Values > 0.70 are generally considered as good (Nunnally and Bernstein, 1994). However, the ICC should be interpreted with the sample variability in mind. Therefore, we calculate the range of scores per item to illustrate the homogeneity of the subjects. Small inter-subject variability results in a depress of the ICC (Weir, 2005). To interpret the values of kappa we used the criteria by Fleiss et al.: values < 0.40 represent poor agreement, 0.41–0.75 fair to good and ≥0.75 represent excellent agreement (Fleiss et al., 2003).

Measurement error, the systematic and random error of an individual patient’s score that is not attributed to true changes in the construct to be measured, was assessed by Bland-Altman plots with the 95% limits of agreement for continuous scores, and the positive and negative percentage agreement for nominal scores.

Scoring the COMPASS

The final version of the COMPASS questionnaire consisted of 15 items. These were divided into two subdomains: external and internal device domains. The external device (speech processor and transmitter) domain and the internal device (receiver/stimulator) domain consisted of seven and eight items, respectively. Items were either multiple choice or visual analogue scale questions. Each item had a maximum score of 5, with a total maximum score of 75. A higher COMPASS score represented a higher awareness level.

Results

Phase 1: Item Generation

Domains of awareness that were identified through literature search were bulging of the implant under the skin, discomfort or pain caused by the implant and sleep disturbances related to the implant. Domains identified through expert interviews were pain caused by the speech processor and transmitter, problems with wearing glasses, satisfaction with the position of the transmitter on the head, and interference of the external implant with daily activities and with wearing head covers (such as helmets). These domains were all mentioned by patients during the interviews in addition to problems with the transmitter coil (magnet falling off or being too strong). These domains of awareness were included in the first draft of the questionnaire. The domains most frequently mentioned were pain caused by the speech processor and/or magnet (mentioned by five out of seven participants), fear or discomfort caused by the external implant falling off the ear, and feeling a bulge where the internal implant resides under the skin (both mentioned by four participants). In order to measure these domains, 18 items were formulated. These items assessed the presence of the domains contributing to awareness and the burden that it created for the patient. Eight dichotomous (yes/no) items assessed the presence of domains; one multiple choice item assessed the ideal position of the transmitter according to the patient; seven visual analogue scale (VAS) items assessed the burden of these domains and two VAS item assessed pain caused by the external parts of the CI and in the area of operation.

Phase 2: Pilot Study (Cognitive Debriefing Test)

A pilot study was conducted with 10 CI patients (see Table 1 for characteristics of the participants). The mean time to complete the questionnaire was 5 min and 21 s (range 3:10–9:40). Based on the results of the item analysis and the cognitive debriefing test small revisions to the questionnaire items and response options were made to ensure comprehensibility and comprehensiveness. Four items measuring interference of the CI with daily activities that overlapped and two items measuring interference of the CI with wearing glasses were fused into two items, one multiple choice item including all activities that the CI could pose troubles with wearing glasses, and one visual analogue scale item measuring burden experienced by these problems. Two items assessing satisfaction with the position of the CI were removed that were deemed not specific for identifying the underlying issue that causes CI awareness. Thus the scoring results of these items would not be helpful for the clinician using this PROM. Two items assessing sleep disturbance caused by the implant were split into four items to increase specificity of the domain by assessing change of sleep position and awareness of the implant while lying on the operated side of the head. Lastly, one item was added to include more complaints other than pain, as suggested by the CI patients. Thus, the number of items was reduced to 15 (see Table 2 and Supplementary Figure 1). Additionally, the lay-out of the paper questionnaire was adapted based on the suggestions of the CI patients.

TABLE 1

Table 1. Characteristics of study participants per study phase.

TABLE 2

Table 2. COMPASS questionnaire items, answer options, and scoring calculations (not original lay-out).

Phase 3: Reliability Study

We included 54 participants in the reliability study. A total of 52 participants (96.3%) filled out and returned both questionnaires. The unilaterally implanted study group had a wide age range (18–82 years) with an average age of 65 years (see Table 1 for demographics of the reliability study participants). Most of the population was male (67.3%). On average, the participants had been using the CI for 30 months (range 3–234 months).

Regarding the reliability analysis, the ICC, which represent reproducibility for the visual analogue scale items, ranged from 0.66 to 0.89 with only one item not meeting the acceptable level of 0.7, namely the item assessing the impact of the transmitter falling off the ear (see Table 3 for all ICC values with 95% confidence intervals). The Cohen’s kappa that was calculated for nominal items ranged from −0.4 to 0.78, with six (sub)items out of 15 scoring above fair to good agreement and five (sub)items scoring excellent agreement. The two multiple choice items (number five and fifteen), contained the four subitems that had poor agreement kappa values, with one subitem on inhibition of work due to the speech processor and transmitter scoring a negative value of −0.40 implying that there was no effective agreement between the two questionnaires on this item (see Table 4 for all Cohen’s kappa values with standard error and 95% confidence intervals).

TABLE 3

Table 3. Reliability and measurement error analysis for visual analogue scale items (continuous data).

TABLE 4

Table 4. Reliability and measurement error analysis for checkbox and multiple choice items (nominal data).

The mean difference for items of continuous variables was −2.18 to 0.22. The 95% limits of agreement (LoA) revealed no statistically significant difference between the two administered questionnaires in all continuous variables (zero is included in each interval) (see Table 3). We observed higher mean differences with wider 95% LoA for items with smaller sample sizes (see Supplementary Figure 2 for Bland-Altman plots). Percentages of agreement ranges between 0 and 95%, and 83.3 and 96.6% for positive and negative agreement, respectively. The positive agreement percentage showed the widest range, with the multiple choice items number three and eight scoring the lowest values (see Table 4).

Discussion

The purpose of the study was to develop and validate a PROM to assess CI awareness, thus the state of mind or situation in which the patient is physically conscious he or she is wearing a cochlear implant and how this consciousness impacts their daily life. The COMPASS questionnaire was developed following the COSMIN guidelines (Mokkink et al., 2010) for development of PROMs and was based on expert opinion and patient interviews, pilot tested with a cognitive interview study, and validated by administrating it to a population of CI recipients. We tested the content validity (comprehensibility, comprehensiveness, and relevance), and reliability of the questionnaire. The COMPASS questionnaire consists of 15 items and showed fair to excellent test-retest reliability for almost all items and measurement error analysis revealed no systematic or random errors of the score per patient. The lowest reliability and positive agreement scores were calculated for the activities impeded by the speech processor and transmitter; in particular work, transport, and social activities. This could suggest that any restrictions caused by the external part of the CI during these particular activities, varies over time, even in the short test-retest time period of 2 weeks.

We believe that prospective assessment of CI awareness using a PROM, can provide more accurate information on any existing problems. We know that hearing aid issues such as discomfort and handling problems, are common amongst users of these medical devices, one study reporting a prevalence of 98% (McCormack and Fortnum, 2013; Bennett et al., 2020). However, some patients might experience problems with their hearing aids, though do not report them to their clinician (Bennett et al., 2019). One study on cochlear implant recipient issues, reported that the majority of patients included in the study (89.8%), had at least one CI device handling problem (Bennett et al., 2017). Previous studies using patient reported outcome measures also found a high prevalence of other adverse events, such as change of taste. Lloyd et al. (2007) and Mikkelsen et al. (2017) reported changes of taste after surgery in 16.9 and 45% of CI patients, respectively. The COMPASS questionnaire could be used by clinicians to assess issues caused by the external and internal components of the CI that contribute to awareness of the cochlear implant. These issues could be solved by counseling or arranging accessories such as an adjustment of magnet power. Moreover, the location of the implant in relation to the ear pinna might be adjusted likewise (cap wearing interferes with superior implant positioning).

The questionnaire fills in the gap and responds to the needs of the implantees that experience negative effects of the presence of the subperiosteal implant. Cochlear implants have undergone tremendous developments in the last decades regarding shape, hardware volume and intrinsic technical refinements. The different manufacturers produce R/S device aspects that are quite divers. One of the interesting developments is the significant reduction in implant volume, that might decrease implant protrusion visible at the level of the skin. Moreover, this might prevent the surgeon to drill a bony well in the temporal cortex as beforehand with the older implant the gold standard has been to drill a well, to tackle this issue. To our knowledge, there is little evidence thus far available regarding the influence of implant volumes reduction or the effects of drilling or not drilling a bony well, on CI awareness of a patient and implant related complaints. Our developed questionnaire meets these goals. Items assessing burden by issues caused by the internal device such as protrusion of the skin, sleep disturbances due to the implant or problems with headgear, could be rectifiable post-implantation by revision surgery (and re-positioning the implant), however, it might be advisable to perform the implantation correctly during primary implantation. Therefore the COMPASS questionnaire could be used in clinic to assess the impact of different surgical methods for positioning and fixation of the R/S device on CI awareness.

A limitation of this study is the study population sample used for development and validation of the questionnaire, which was recruited from a single center. This could introduce selection bias, however, participants were operated by several CI surgeons with different surgical techniques. Also, assessment of the criterion validity of the COMPASS questionnaire could not be executed. After extensive literature research, we were unable to find validated outcome measures assessing CI use as defined in this study. Furthermore, despite our hypothesis that there are indeed differences of CI awareness between groups, it was impossible to execute this validation step. We expect that patients operated with different fixation techniques of the R/S device will differ in CI awareness. However, in our center we only use one fixation technique (the bony bed technique), and thus we could not compare these groups. Lastly, all four CI device brands were represented in the study population, and patients included in the study had sufficient experience with using the CI to contribute to the study.

In conclusion, the COMPASS questionnaire has good reliability and validity. Combining this PROM with clinical findings may assist in the routine follow up of patients with CI. Furthermore, it can be used as an endpoint in a clinical study, to evaluate different surgical techniques and its effect on awareness.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the local ethical committee (Institutional Review Board of the UMC Utrecht) (METC protocol 19-722/C). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

All authors contributed equally to draft and revised the manuscript and approved the final submission.

Funding

Oticon Medical directly funded the Ph.D. research project of LM, via the University Medical Center Utrecht.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank the audiologists, speech therapists, and patients who participated in this study for their input to the questionnaire.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2022.830768/full#supplementary-material

References

Bennett, R. J., Jayakody, D. M. P., Eikelboom, R. H., and Atlas, M. D. (2017). Self-reported cochlear implant management skills: development and validation of the self-administered Cochlear Implant Management Skills (CIMS-self) survey. Clin. Otolaryngol. 42, 164–171. doi: 10.1111/coa.12713

PubMed Abstract | CrossRef Full Text | Google Scholar

Bennett, R. J., Kosovich, E. M., Stegeman, I., Ebrahimi-Madiseh, A., Tegg-Quinn, S., and Eikelboom, R. H. (2020). Investigating the prevalence and impact of device-related problems associated with hearing aid use. Int. J. Audiol. 59, 615–623. doi: 10.1080/14992027.2020.1731615

PubMed Abstract | CrossRef Full Text | Google Scholar

Bennett, R. J., Laplante-Lévesque, A., and Eikelboom, R. H. (2019). How do hearing aid owners respond to hearing aid problems? Ear Hear. 40, 77–87. doi: 10.1097/AUD.0000000000000595

PubMed Abstract | CrossRef Full Text | Google Scholar

Carlson, M. L., Driscoll, C. L. W., Gifford, R. H., and McMenomey, S. O. (2012). Cochlear implantation: current and future device options. Otolaryngol. Clin. North Am. 45, 221–248. doi: 10.1016/j.otc.2011.09.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Fleiss, J. L., Levin, B., and Paik, M. C. (2003). Statistical Methods for Rates and Proportions. Hoboken, NJ: John Wiley & Sons.

Google Scholar

Gagnon, E. B., Eskridge, H., and Brown, K. D. (2020). Pediatric cochlear implant wear time and early language development. Cochlear Implants Int. 21, 92–97. doi: 10.1080/14670100.2019.1670487

PubMed Abstract | CrossRef Full Text | Google Scholar

Gaylor, J. M., Raman, G., Chung, M., Lee, J., Rao, M., Lau, J., et al. (2013). Cochlear implantation in adults: a systematic review and meta-analysis. JAMA Otolaryngol. Head Neck Surg. 139, 265–272.

Google Scholar

Hinderink, J. B., Krabbe, P. F., and Van Den Broek, P. (2000). Development and application of a health-related quality-of-life instrument for adults with cochlear implants: the Nijmegen cochlear implant questionnaire. Otolaryngol. Head Neck Surg. 123, 756–765.

Google Scholar

Holder, J. T., Dwyer, N. C., and Gifford, R. H. (2020). Duration of processor use per day is significantly correlated with speech recognition abilities in adults with cochlear implants. Otol. Neurotol. 41, e227–e231. doi: 10.1097/MAO.0000000000002477

PubMed Abstract | CrossRef Full Text | Google Scholar

Lloyd, S., Meerton, L., Di Cuffa, R., Lavy, J., and Graham, J. (2007). Taste change following cochlear implantation. Cochlear Implants Int. 8, 203–210. doi: 10.1179/cim.2007.8.4.203

PubMed Abstract | CrossRef Full Text | Google Scholar

Loeffler, C., Aschendorff, A., Burger, T., Kroeger, S., Laszig, R., and Arndt, S. (2014). Quality of life measurements after cochlear implantation. Open Otorhinolaryngol. J. 4, 47–54.

Google Scholar

McCormack, A., and Fortnum, H. (2013). Why do people fitted with hearing aids not wear them? Int. J. Audiol. 52, 360–368. doi: 10.3109/14992027.2013.769066

PubMed Abstract | CrossRef Full Text | Google Scholar

McRackan, T. R., Bauschard, M., Hatch, J. L., Franko-Tobin, E., Droghini, H. R., Nguyen, S. A., et al. (2018). Meta-analysis of quality-of-life improvement after cochlear implantation and associations with speech recognition abilities. Laryngoscope 128, 982–990. doi: 10.1002/lary.26738

PubMed Abstract | CrossRef Full Text | Google Scholar

Mikkelsen, K. S., Ovesen, T., and Swan, C. Z. (2017). Pre- and post-operative dizziness, tinnitus, and taste disturbances among cochlear implant recipients. J. Laryngol. Otol. 131, 309–315. doi: 10.1017/S0022215116010008

PubMed Abstract | CrossRef Full Text | Google Scholar

Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., et al. (2010). The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual. Life Res. 19, 539–549. doi: 10.1007/s11136-010-9606-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Nunnally, J., and Bernstein, I. (1994). Psychometric Theory, 3rd Edn. New York, NY: McGraw-Hill.

Google Scholar

Qin, S., Nelson, L., McLeod, L., Eremenco, S., and Coons, S. J. (2019). Assessing test–retest reliability of patient-reported outcome measures using intraclass correlation coefficients: recommendations for selecting and documenting the analytical formula. Qual. Life Res. 28, 1029–1033. doi: 10.1007/s11136-018-2076-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J. Strength. Cond. Res. 19, 231–240. doi: 10.1519/15184.1

PubMed Abstract | CrossRef Full Text | Google Scholar