- 1Department of Occupational Therapy, School of Health Science, International University of Health and Welfare, Narita, Japan
- 2Department of Occupational Therapy, Graduate School of Human Health Science, Tokyo Metropolitan University, Arakawa, Japan
Introduction: Recently, a method was developed to predict the motor Functional Independence Measure (FIM) score at discharge in patients with stroke by stratifying the effects of factors such as age and cognitive function and multiplying those by the influence coefficients of these factors. However, an evaluation of the predictive performance of the method is required for clinical application. The present study aimed to evaluate the predictive performance of this prediction method.
Methods: Patients with stroke discharged from a rehabilitation ward between April 2021 and September 2022 were included. Predicted values of the motor FIM score at discharge were calculated after data collection from the hospital’s patient database. The concordance between predicted and actual values was evaluated using the interclass correlation coefficient; moreover, the residual values were calculated.
Results: In total, 207 patients were included in the analysis. The median age was 79 (69–85) years, and 112 (54.1%) patients were male. The interclass correlation coefficient between predicted and actual values was 0.84 (95% confidence interval 0.75–0.89) for the motor FIM score at discharge. Meanwhile, the median residual value was 5.3 (−2.0–10.3) for the motor FIM score at discharge.
Discussion: The prediction method was validated with good performance. However, the residual values indicated that some cases deviated from the prediction. In future studies, it will be necessary to improve the predictive performance of the method by clarifying the characteristics of cases that deviate from the prediction.
1 Introduction
Stroke is a leading cause of disability. Long-term disabilities following a stroke may cause significant physical, mental, and financial burdens for patients and their families (1); moreover, stroke is a major health problem worldwide. In Japan, more than 1.1 million people suffer from stroke each year (2), and it is the second leading cause (only behind dementia) of long-term care requirement. Promoting independent living among patients with stroke remains an important social issue. Stroke rehabilitation is believed to play an important role in solving this social problem by promoting patients’ independent living.
Prognostic prediction is important for efficient and effective stroke rehabilitation (3). Accurate prognostic prediction is essential for setting attainable goals, providing information to patients and relatives, and making shared decisions (4). In particular, prognosis prediction of activities of daily living (ADL) has been of interest among clinicians and therapists because ADL dependency is associated with discharge destination and care burden.
Previous studies have attempted to predict the Functional Independence Measure (FIM) score, an ADL assessment scale, in patients with stroke using various methods (4–7). In Japan, several studies have used multiple regression analysis to predict the motor FIM (m-FIM) score at discharge, based on data such as the FIM score at admission and age, in patients with stroke in specialized rehabilitation wards (8–11). Multiple regression analysis is considered useful when there is a linear relationship between explanatory and objective variables. However, there is no linear relationship between the m-FIM score at admission or age and the m-FIM score at discharge (12). Therefore, the stratification of factors affecting the outcomes may be a useful method for a more accurate prediction of patient prognosis.
Recently, a novel prediction method that predicts the m-FIM score at discharge by stratifying the effects of factors such as age, cognitive function at admission, and duration from onset to admission and multiplying the influence coefficients of these factors was developed in Japan (12). This prediction equation consists of four variables routinely collected in rehabilitation wards in Japan, making it highly applicable in clinical practice. However, no study has examined the concordance between the predicted and actual values in this prediction method; in addition, its potential for clinical application has not been fully validated. The present study, therefore, aimed to evaluate the predictive performance of the prediction method in a validation cohort.
2 Materials and methods
2.1 Participants and setting
Patients with stroke diagnosed with cerebral infarction or hemorrhage and admitted to a specialized rehabilitation ward in a subacute care hospital between April 2021 and September 2022 were included in this retrospective study. The hospital is located in the northwestern part of Tokyo metropolitan area and has a total of 150 beds. All the patients with stroke underwent rehabilitation therapy every day during hospitalization. The rehabilitation programs included physical, occupational, and speech therapies, as necessary. Patients received a maximum of 3 h of rehabilitation therapy per day.
The exclusion criteria were the same as those in the development study (12): (1) patients with subarachnoid hemorrhage; (2) patients admitted within 7 days or > 60 days after onset; (3) patients who spent <14 days or > 180 days in the hospital; (4) patients transferred to acute care hospitals; (5) patients with medical conditions that worsened during hospitalization; and (6) patients with an m-FIM score of 91 points on admission.
This study was approved by the Ethics Committee of Tokyo Metropolitan University (approval no. 22066) and conducted in accordance with the “Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis” statement (13).
Owing to the retrospective design of the study, the participants were recruited on an opt-out method in lieu of informed consent. Data were accessed for research purposes from 20 December 2022 to 31 March 2023. During data collection, the first author had access to information that could identify individual participants.
2.2 Outcome measure
The FIM (14) was used to measure study outcomes. It is one of the most widely used tools for assessing ADL dependence in patients with disabilities and has proven to have a high reliability (15). The FIM consists of 18 items, including feeding, toileting, and communication. Each item is scored on a seven-point scale from 1 to 7, with higher scores indicating greater independence in ADL. Total FIM scores range from 18 to 126, m-FIM scores range from 13 to 91, and cognitive FIM (c-FIM) scores range from 5 to 35. The FIM scoring was done via a discussion between well-trained nurses and occupational therapists assigned to each patient. In the present study, m-FIM and c-FIM scores at admission and discharge were collected.
2.3 Variables
Demographic and clinical data necessary for prediction were collected from the hospital’s patient database. As in the development study (12), the following variables were investigated: age, sex, stroke type, duration from onset to admission (transfer interval), and length of hospital stay.
2.4 Prediction method
The prediction method was as previously described (12). The first step in the prediction method is to calculate the standard value of the m-FIM gain from the m-FIM score at admission based on Figure 1A. The standard value of the m-FIM gain was the median m-FIM gain calculated for each group after classification of the participants in the previous study into 13 groups with the m-FIM score at admission in six-point increments. The standard value for m-FIM gain was calculated as 35 if the patient’s m-FIM score at admission was 32. The next step was to calculate each effect coefficient from age, cognitive function, and transfer interval according to Figures 1B–D. Finally, the predictive value of the m-FIM score at discharge was calculated using the following formula:
Figure 1. Standard values of motor Functional Independence Measure (FIM) gain and the influence coefficients for age, cognitive function, and transfer interval. (A) Standard values of motor FIM gain in 13 groups stratified by motor FIM score at admission. (B) Influence coefficient for age. (C) Influence coefficient for cognitive function. (D) Influence coefficient for transfer interval Reprinted from Tokunaga et al. (12).
For example, if the m-FIM score at admission was 28, age was 87 years, c-FIM score at admission was 12, and number of days from onset to admission was 40, the predictive value of the m-FIM score at discharge would be (28 + 34) × 0.920 × 1.006 × 0.988 = 56.7.
2.5 Statistical analyses
Statistical analyses were performed using IBM SPSS Statistics for Windows, version 28.0. (IBM Corp., Armonk, N.Y., United States). Statistical significance was set at p < 0.05. Descriptive statistics were reported for the demographic data of the participants. Data are reported as means and standard deviation or medians with interquartile range for continuous variables, according to normality. Categorical variables are reported as percentages. In addition, one-sample Wilcoxon and chi-square tests were performed using the data from the previous study as comparison values and compared with the data from the present study; the one-sample test is a useful method for comparing representative values of samples from a previous study with the data in the present study.
After calculating the predicted values of m-FIM score at discharge, interclass correlation coefficients (ICC [2.1]) with 95% confidence intervals (CIs) were determined to reveal the concordance between predicted and actual values. The ICC is a valid measure for assessing the degree of concordance when the same participants are measured in different ways; it has been used in previous studies to assess the performance of prediction models for FIM scores (16, 17). The minimum sample size required to detect an ICC of 0.6 with an expected precision of 0.1 and a CI of 95% was 159 (18). In addition, the residual values were calculated by subtracting the predicted values from the actual values.
Subsequently, to clarify the characteristics of cases that deviated from the predicted values, the characteristics were compared among three groups: (1) a group within ±10 points of the residual m-FIM score at discharge, (2) a group with a residual > 10 points, and (3) a group with a residual <−10 point. In a previous study, an m-FIM score < 50 points was defined as complete assistance, 50–69 points as incomplete assistance, ≥70 points as self-care independence, and ≥ 80 points as walking independence (19). In other words, a patient’s level of ADL independence was considered to change approximately every 10 points. Therefore, in this study, a case was defined as deviating from the prediction if the error between the predicted and actual values was >10 points. Therefore, the group with residuals within ±10 points was defined as “the group within the predicted range.” In addition, the group with residuals > 10 points was defined as “the group above the prediction” and the group with residuals < −10 points was defined as “the group below the prediction.” The Kruskal–Wallis test was used to compare the three groups, followed by the Dunn’s test for multiple comparisons.
3 Results
3.1 Participants characteristics
In total, 274 patients with stroke were admitted during the study period. Of these, 67 patients were excluded: 20 were admitted within 7 days or > 60 days after onset, 19 were transferred to acute care hospitals, 15 had subarachnoid hemorrhage, eight spent < 14 days or > 180 days in the hospital, and five had medical conditions that worsened during hospitalization. No patients had missing data. Therefore, 207 patients were finally included in the analysis (Figure 2).
Demographic variables are presented in Table 1. The median age was 79 (69–85) years, and 112 patients (54.1%) were male. One hundred and forty-six patients (70.5%) were diagnosed with cerebral infarction and 61 (29.5%) with hemorrhage. On admission, the median m-FIM score was 36 (20–53) and median c-FIM score was 18 (14–22); on discharge, the median m-FIM score was 73 (41–85) and median c-FIM score was 28 (18–33). In addition, the median m-FIM gain was 29 (14–34) and the median c-FIM gain was 7 (3–12).
Table 1. Comparison of demographic and clinical characteristics between the development and validation groups.
Compared to the development group, the validation group was significantly older, had a larger proportion of women, and had a longer transfer interval and length of stay (p < 0.001). Furthermore, the validation group had significantly lower m-FIM and c-FIM scores at admission and discharge but significantly higher FIM gains (p < 0.001).
3.2 Concordance and residuals between predicted and actual values in m-FIM scores at discharge
Figure 3 is a scatter plot showing the relationship between predicted and actual values in this prediction model. For the concordance between predicted and actual m-FIM scores at discharge, the ICC [2.1] was 0.84 (95% CI 0.75–0.89). Meanwhile, the median residual was 5.3 (−2.0–10.3).
Figure 3. Relationship between the predicted and actual values for motor Functional Independence Measure (FIM) score at discharge, and the median residuals (n = 207). FIM, Functional Independence Measure; ICC, interclass correlation coefficient.
3.3 Comparison of the three groups classified by residuals in the m-FIM scores at discharge
The three groups were classified according to residuals in the m-FIM scores at discharge; the results of the comparison of demographics and clinical data at admission are shown in Table 2. Only the m-FIM score at admission was significantly different between the three groups (p = 0.01). In multiple comparisons, the m-FIM scores at admission were significantly higher in the group within the predicted range than in the group above the prediction (p = 0.03). No significant differences were found between the other groups (p ≥ 0.05).
4 Discussion
This study examined the predictive performance of a method for predicting the m-FIM scores at discharge using the standard values of FIM gain and influence coefficients for age, cognitive function, and transfer interval. The prediction model showed good predictive performance, as it exhibited high concordance between predicted and measured values.
Comparing the characteristics of the validation group in the present study with those of the development group in the previous study (12), there were significant differences in age, gender, transfer interval, length of hospital stay, stroke type, m-FIM scores at admission and discharge, and FIM gain. Despite these differences, the ICC [2.1] between the predicted and actual values was 0.84 (95% CI 0.75–0.89) in the validation group. Regarding ICC values, <0.50 indicate poor validity, 0.50–0.75 indicate moderate validity, 0.75–0.90 indicate good validity, and > 0.90 indicate excellent validity (20). These results indicate that the predictive performance of this prediction method is good.
Various methods have been reported to predict m-FIM scores at discharge in patients with stroke. Wada et al. tested the predictive accuracy in models using the reciprocal of m-FIM at admission and m-FIM effectiveness, and reported that the ICCs were 0.90 and 0.89 for each model (16). Kimura et al. (17) reported an ICC of 0.89 for their predictive model using a logarithmic model. However, using the above method would require collecting more data or data at two time points and complex calculations. The method investigated in this study has similar predictive accuracy to the method described above; however, our method is relatively simple, consisting of four variables that can be easily collected in a clinical setting. Therefore, this prediction method is superior in terms of ease of use in a clinical setting.
In contrast, the median residuals were 5.3 (−2.0–10.3) for the m-FIM score at discharge. The median residuals reported in the development study were 0 for the m-FIM score at discharge (12), and our results tended to show values higher than these. This suggests that some cases in the present study deviated from the predictions.
Therefore, to clarify the characteristics of the cases that deviated from the predictions at the time of admission, we compared the cases in three groups classified according to the residual in m-FIM scores at discharge. The results showed no significant differences in age, sex, stroke type, duration from onset to admission, and the c-FIM score at admission. Only the m-FIM score at admission showed a significant difference, being significantly higher in the group within the predicted range than in the group above the predicted range. However, there were no significant differences among the other groups. Therefore, it was difficult to clearly distinguish between groups that were within the predicted range and those that deviated from the predictions from the data collected at admission in this study.
Various factors have been reported to be associated with improved ADLs, including premorbid physical activity (21), lower severity of paresis (22), self-efficacy (23), improved nutrition during hospitalization (24), energy intake at admission (24), and interventions such as motor rehabilitation (3). Future studies on sensitivity and specificity are needed to identify patient populations with a greater predictive power, based on data collected from diverse perspectives.
This study had some limitations. First, it was performed at a single center in Japan; therefore, it is difficult to apply our results to cases from different regions and cultural backgrounds. The applicability of this predictive model in cases from different regions and cultural backgrounds needs to be verified in future studies. Second, only a few variables were employed in this prediction method. Future studies are needed to examine the impact of variables not included in the models, such as history of chronic disease and the amount and types of rehabilitation interventions, on the accuracy of outcome prediction; in addition, future studies attempting to update the models are required. Third, because this was a retrospective observational study, the limited variables collected at admission did not allow for adequate analysis of the groups for which this predictive tool would have high a predictive power. In the future, multidimensional data should be collected to clearly define the criteria for application of this tool.
This prediction method, as previously mentioned, has several challenges; however, the results of this study showed that the model demonstrated good predictive performance. The prediction method consists of four parameters that are easy to collect in routine clinical practice; therefore, we expect increased application of the prediction method in clinical practice in the future.
5 Conclusion
This study evaluated the prediction performance of a method to predict the m-FIM score at discharge by stratifying the effects of factors such as age, cognitive function at admission, and duration from onset to admission and then multiplying by influence coefficients of these factors. The results of this study showed high concordance between the predicted and actual values of the m-FIM scores at discharge. Meanwhile, the median residuals were 5.3 (−2.0–10.3) for the m-FIM score at discharge, suggesting that a certain number of patients deviated from the predictions. In future studies, clarifying the characteristics of cases that deviate from the predicted values is expected to further clarify the criteria for the application of this prediction method and improve its prediction accuracy.
Data availability statement
The data analyzed in this study is subject to the following licenses/restrictions: data are not authorized for release to the public by the Ethics Committee because they contain potentially personally identifiable patient information. Requests to access these datasets should be directed to the Ethics Committee of Tokyo Metropolitan University: a-rinri@jmj.tmu.ac.jp.
Ethics statement
The studies involving humans were approved by the Ethics Committee of Tokyo Metropolitan University (approval No. 22066). The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participants’ legal guardians/next of kin because the participants in this study were discharged patients and the study design was to analyze existing information, they were recruited on an opt-out basis.
Author contributions
RK: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Validation, Visualization, Writing – original draft, Writing – review & editing. NK: Conceptualization, Project administration, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work was supported by JSPS KAKENHI (grant number JP23K19808) from the Japan Society for the Promotion of Science KAKENHI (recipient: RK).
Acknowledgments
The authors would like to thank all colleagues and participants who contributed to this study.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Lin, WY, Chen, CH, Tseng, YJ, Tsai, YT, Chang, CY, Wang, HY, et al. Predicting post-stroke activities of daily living through a machine learning-based approach on initiating rehabilitation. Int J Med Inform. (2018) 111:159–64. doi: 10.1016/j.ijmedinf.2018.01.002
2. Ministry of Health, Labour and Welfare in Japan . Number of patients with cerebrovascular disease (in Japanese). (2024). Available online at: https://www.mhlw.go.jp/stf/wp/hakusyo/kousei/18/backdata/01-01-02-04.html. (Accessed 17 April 2024).
3. Kwakkel, G, Stinear, C, Essers, B, Munoz-Novoa, M, Branscheidt, M, Cabanas-Valdes, R, et al. Motor rehabilitation after stroke: European stroke organisation (ESO) consensus-based definition and guiding framework. Eur Stroke J. (2023) 8:880–94894. doi: 10.1177/23969873231191304
4. Scrutinio, D, Lanzillo, B, Guida, P, Mastropasqua, F, Monitillo, V, Pusineri, M, et al. Development and validation of a predictive model for functional outcome after stroke rehabilitation: the Maugeri model. Stroke. (2017) 48:3308–15. doi: 10.1161/STROKEAHA.117.018058
5. Koyama, T, Matsumoto, K, Okuno, T, and Domen, K. A new method for predicting functional recovery of stroke patients with hemiplegia: logarithmic modelling. Clin Rehabil. (2005) 19:779–89. doi: 10.1191/0269215505cr876oa
6. Sonoda, S, Saitoh, E, Nagai, S, Okuyama, Y, Suzuki, T, and Suzuki, M. Stroke outcome prediction using reciprocal number of initial activities of daily living status. J Stroke Cerebrovasc Dis. (2005) 14:8–11. doi: 10.1016/j.jstrokecerebrovasdis.2004.10.001
7. Inouye, M . Predicting outcomes of patients in Japan after first acute stroke using a simple model. Am J Phys Med Rehabil. (2001) 80:645–9. doi: 10.1097/00002060-200109000-00003
8. Sato, A, Fujita, T, Ohashi, Y, Yamamoto, Y, Suzuki, K, and Otsuki, K. A prediction model for activities of daily living for stroke patients in a convalescent rehabilitation ward. J Allied Health Sci. (2016) 7:1–6. doi: 10.15563/jalliedhealthsci.7.1
9. Jeong, S, Inoue, Y, Kondo, K, Matsumoto, D, and Shiraishi, N. Formula for predicting FIM for stroke patients at discharge from an acute ward or convalescent rehabilitation ward. Jpn J Compr Rehabil Sci. (2014) 5:19–25. doi: 10.11336/jjcrs.5.19
10. Tokunaga, M, and Sannomiya, K. Explanatory variables to use in a multiple regression analysis to predict stroke patients’ motor FIM score at discharge from convalescent rehabilitation wards: an investigation of patients with a motor FIM score of less than 40 points at admission. Jpn J Compr Rehabil Sci. (2020) 11:102–8. doi: 10.11336/jjcrs.11.102
11. Tokunaga, M, Watanabe, S, and Sonoda, S. A method of calculating functional Independence measure at discharge from functional Independence measure effectiveness predicted by multiple regression analysis has a high degree of predictive accuracy. J Stroke Cerebrovasc Dis. (2017) 26:1923–8. doi: 10.1016/j.jstrokecerebrovasdis.2017.06.028
12. Tokunaga, M, Sannomiya, K, Nakashima, Y, Nojiri, S, Tokisato, K, Katsura, K, et al. Formula for predicting FIM gain and discharge FIM—methods using median values of FIM gain stratified by admission FIM, age, cognitive function, and transfer interval―. Jpn J Compr Rehabil Sci. (2015) 6:6–13. doi: 10.11336/jjcrs.6.6
13. Collins, GS, Reitsma, JB, Altman, DG, and Moons, KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. (2015) 162:55–63. doi: 10.7326/M14-0697
14. Granger, CV, Hamilton, BB, Linacre, JM, Heinemann, AW, and Wright, BD. Performance profiles of the functional independence measure. Am J Phys Med Rehabil. (1993) 72:84–9. doi: 10.1097/00002060-199304000-00005
15. Ottenbacher, KJ, Hsu, Y, Granger, CV, and Fiedler, RC. The reliability of the functional independence measure: a quantitative review. Arch Phys Med Rehabil. (1996) 77:1226–32. doi: 10.1016/s0003-9993(96)90184-7
16. Wada, Y, Sonoda, S, Okamoto, S, Watanabe, M, Okazaki, H, and Okuyama, Y. Comparison of prediction accuracy of the total score of FIM motor items at discharge in poststroke patients in a Kaifukuki rehabilitation ward. Jpn J Compr Rehabil Sci. (2019) 10:71–6. doi: 10.11336/jjcrs.10.71
17. Kimura, Y, Yamada, M, Hamanaka, K, Tanaka, N, and Muroh, Y. Usefulness of the prediction method based on a logarithmic model for functional recovery in stroke patients: in case of using the motor-functional Independence measure score. Int J Rehabil Res. (2017) 40:134–7. doi: 10.1097/MRR.0000000000000219
18. Bonett, DG . Sample size requirements for estimating intraclass correlations with desired precision. Stat Med. (2002) 21:1331–5. doi: 10.1002/sim.1108
19. Tsuji, T, Sonoda, S, and Chino, N. The ADL structure for stroke patients at admission and discharge based on the functional Independence measure (in Japanese). Jpn J Rehabil Med. (1996) 33:301–9. doi: 10.2490/jjrm1963.33.301
20. Koo, TK, and Li, MY. A guideline of selecting and reporting Intraclass correlation coefficients for reliability research. J Chiropr Med. (2016) 15:155–63. doi: 10.1016/j.jcm.2016.02.012
21. Ursin, MH, Ihle-Hansen, H, Fure, B, Tveit, A, and Bergland, A. Effects of premorbid physical activity on stroke severity and post-stroke functioning. J Rehabil Med. (2015) 47:612–7. doi: 10.2340/16501977-1972
22. Veerbeek, JM, Kwakkel, G, van Wegen, EE, Ket, JC, and Heymans, MW. Early prediction of outcome of activities of daily living after stroke: a systematic review. Stroke. (2011) 42:1482–8. doi: 10.1161/STROKEAHA.110.604090
23. Szczepańska-Gieracha, J, and Mazurek, J. The role of self-efficacy in the recovery process of stroke survivors. Psychol Res Behav Manag. (2020) 13:897–906. doi: 10.2147/PRBM.S273009
Keywords: stroke, activities of daily living, functional independence measure, prediction, predictive performance, influence coefficient
Citation: Kobayashi R and Kobayashi N (2024) Performance of a prediction method for activities of daily living scores using influence coefficients in patients with stroke. Front. Neurol. 15:1419405. doi: 10.3389/fneur.2024.1419405
Edited by:
Francesco Corea, Azienda USL Umbria 2, ItalyReviewed by:
Yusuke Masuda, Shonan University of Medical Sciences, JapanAna Isabel Fumagalli, Sanatorio Parque Rosario Argentina, Argentina
Copyright © 2024 Kobayashi and Kobayashi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ryu Kobayashi, ryuryukobkob@gmail.com