Interrater reliability of the BelRAI Social Supplement in Flanders, Belgium: Simultaneous rating of community-dwelling adults with care needs during COVID-19

Background: The BelRAI Screener is a short-form assessment consolidating internationally validated interRAI items focusing on physical and psychological aspects of functioning and problems with activities of daily living. It was fully implemented in the Flemish home care setting as of June 2021. In a biopsychosocial model for developing a personalized and effective care plan social and contextual aspects are considered equally important to biomedical ones. Thus, a social supplement to the BelRAI Screener was collaboratively developed with stakeholders and tested to gather additional information on the social context of community-dwelling adults with care needs.

Objective and methods: To examine the interrater reliability of the BelRAI Social Supplement in Flanders, Belgium, an observational study was conducted using a convenience sample. The method of simultaneous rating was used due to strict COVID-19 guidelines at the time and to minimize assessment burden. Fifty two community-dwelling adults requesting home care support were simultaneously assessed by two independent assessors during home visits. Interrater reliability was tested on all 80 items of the BelRAI Social Supplement using observed agreement, kappa coefficients, and intraclass correlation coefficients.

Results: The kappa mean (0.74) and median (0.79) values for nominal items, show substantial agreement, while the kappa mean and median values for ordinal items were 0.81 and 0.90, which correspond to almost perfect agreement. Following the traditional cut-off points for the interpretation of the kappa statistic, reliability was almost perfect (κ > 0.81) for 49% of all items, substantial (0.60 < κ ≤ 0.80) for 33%, moderate (0.40 < κ ≤ 0.60) for 8%, and poor (κ < 0.40) for 10%. The majority of items with poor kappa value, showed a high observed agreement, reflecting homogeneity of the sample rather than poor agreement.

Conclusion: The strength of kappa agreement for the items in this version of the BelRAI Social Supplement is generally substantial to almost perfect, with high proportions of observed agreement. COVID-19 restrictions had a large impact on the planning and execution of the home visits. A final optimization of the instrument and accompanying manual according to the findings will result in an improved version ready for nation-wide implementation.

Introduction

The aging of populations is a long-term trend which significantly strains the capacity and performance of regional and national health and social systems. This growing share of the elderly population is associated with an increase in demand for health and social care services, including the use of social care at home (Tarricone and Tsouros, 2008; Hood et al., 2022). Along with an increase in the number of older persons with care needs staying in their communities, we see a rise in the concurrence of comorbidities and in complexity of care needs (Fried et al., 2004; Valderas et al., 2009). These converging trends result in a pressing need for an update of the (mostly fragmented) health care systems, in which care providers, interventions and policies focus on one aspect of a person’s complex care needs (North, 2020). The time and resources constraints due to this fragmentation obstruct the process of assessing the complexity of a person’s health-related and contextual characteristics (Whiteneck et al., 2004; North, 2020).

In 2016, the World Health Organization (WHO) developed a framework on integrated people-centered health services that highlights a holistic and multi-disciplinary outpatient approach to caregiving (World Health Organization [WHO], 2015). In this framework, a person with care needs is at the center of a network that consists of their family and close community. This people-centered approach acknowledges the close bond between the individual and their social and cultural capital, with persons and their communities considered to be active participants in health and social care services. The health services within this framework strive toward providing holistic care (i.e., treating not just the disease, but the whole person), using an out-patient approach (i.e., treatment does not require hospitalization), and is accessible across different levels of health care delivery (Kraus de Camargo, 2011; van Dulmen et al., 2015).

A standardized way of holistically assessing a person with disabilities—and sharing those findings with other care professionals—is an important step toward translating and implementing the WHO-model in practice (Berg et al., 2009; Van Doren et al., 2021a). Comprehensive care assessments, such as the interRAI suite of instruments, provide professionals with a standardized way to evaluate a person’s functional status and social context characteristics (Heckman and Jónsson, 2018).

The interRAI suite of instruments is internationally validated and contains a variety of comprehensive assessment tools developed with a particular setting or population in mind (home care, long-term care facilities, mental health care, palliative care, etc.) (Gray et al., 2009). Contrary to other disease- or population-specific assessment tools, all interRAI instruments share a set of core items and terminology. By providing a universal language to the people working in different health and social services settings, the transfer of information between health professionals, administrators, and governments improves. This facilitates multi- and interdisciplinary assessments, providing person-oriented care and making informed decisions at the policy-level (Carpenter and Hirdes, 2013; Hirdes et al., 2020).

In 2018, the Belgian government opted for a mandated implementation of the interRAI instruments in all care sectors in order to translate the WHO’s framework in practice (Vandeurzen, 2018; World Health Organization [WHO], 2019). This process created a demand for the cultural adaptation and the translations of the interRAI instruments into Dutch, French and German, the three national languages of Belgium. The collection of these translated instruments is called BelRAI (Declercq et al., 2009).

Since June 2021, the first BelRAI instrument—the BelRAI Screener–was fully implemented in the home care setting in Flanders (i.e., the Dutch-speaking region of Belgium) (Vermeire, 2020). The BelRAI Screener is a short-form assessment consolidating internationally validated interRAI items focusing on physical and different aspects in the biopsychosocial model of disability (Engel, 1980; Borrell-Carrió et al., 2004). This Screener allows professionals to obtain a rapid and standardized first assessment of a person’s care needs and gauge whether someone needed a full BelRAI Home Care assessment. The assessment also allows for the calculation of a dependency and care complexity index to checks a person’s eligibility for a regional care budget (Vermeulen et al., 2015; Van Doren et al., 2021b). As of 2023, the BelRAI Home Care instrument will be implemented in home care and the BelRAI Long-Term Care Facilities instrument will be implemented in nursing homes.

Although the BelRAI Screener is positively evaluated by professionals, especially those active in social care mention the lack of socially oriented items (Vermeulen et al., 2015; Vernimmen et al., 2018). The context a person lives in has an impact on their functioning as well. To answer this question, a BelRAI Social Supplement was developed. This Social Supplement is implemented in the whole of Flanders as of June 2022. While the collaborative development and extensive testing of the BelRAI Social Supplement previously were described (Van Doren et al., 2021a,2022), testing the interrater reliability of the instrument had not been done yet.

Evaluating the reliability of the assessment and quality of the data is important for all users of the assessments and their output (Wellens et al., 2012a,b; Van Doren et al., 2021b). One way of evaluating an instrument is by analyzing the consistency in coding between two raters assessing the same person in a short time span. Low interrater reliability endangers the validity of the assessment, as it is important that the results are reproducible and consistent between two assessors. In the past, there have been a handful of international studies evaluating the interrater reliability of several interRAI instruments (Martin et al., 2007; Gray et al., 2008; Hirdes et al., 2008; Wellens et al., 2012c; Kim et al., 2015; Boscart et al., 2021). Based on these results, the respective instruments, manuals and training materials and their translations have been updated and improved. The purpose of this study is thus to test the interrater reliability of the BelRAI Social Supplement among persons requesting home care in Flanders, Belgium. This study was a collaborative effort with the Flemish Government and partners in the field.

Materials and methods

An observational study was conducted using a convenience sample of persons seeking home care support in Flanders, the northern region of Belgium. Community-dwelling adults with care needs (physical or mental) or disabilities are included. Furthermore, the person had to be able to speak the native language (Dutch) and give its consent for an assessment. Persons applying for maternity care or other short-term care and support were excluded. To determine interrater reliability, two independent assessors completed the BelRAI Social Supplement for the same person with a care need.

Gwet (2014) provides an overview of the minimum number of assessments needed to achieve a desired confidence interval for interrater reliability studies. We chose to aim for a confidence interval of 85% in this study, for which Gwet (2014) states that we would need to complete at least 44 assessments. This seemed a feasible number in the context of this study.

With regards to the recruitment of social care workers, we selected a limited number of social care workers who had been trained in completing the BelRAI Screener as well as the Social Supplement, and who had previous experience with administering both during a home visit. We created a directory of all eligible assessors so that home care services were able to contact the appropriate candidates within their own organizations. Interested assessors were in turn asked to contact us to receive further instructions and schedule their usual home visits on the approved dates. All participating assessors (n = 9) needed to attend an (online) refreshment course of about 3 h in total with information about completing the BelRAI Social Supplement and the practical arrangements.

One trained social care worker and one of the researchers (SVD)¹ simultaneously completed the BelRAI Social Supplement during a home visit according to standard interRAI conventions. The evaluation was based on a semi-structured interview and observations made during the conversation. Both assessors were blinded to the others’ results and were not permitted to discuss the case with each other, nor to exchange information after the home visit.

During the home visit, the social care worker (rater one) guided the home visit to provide the appropriate services and to complete the BelRAI Screener and the BelRAI Social Supplement. The second rater (SV) shadowed the social worker and simultaneously completed the BelRAI Social Supplement based on the conversation and their own observations. The second rater remained in the background during the assessment.

We reported this study in accordance with the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) to make interpretation and synthesis of other reliability studies possible (Kottner et al., 2011).

Ethical approval and consent to participate

We obtained ethics approval to conduct this study from the Social and Societal Ethics Committee of KU Leuven (file number G- 2019 05 1654). All potential participants were provided with full information about the study during the scheduling process of the home visit. During field testing, all persons receiving or requesting social care services, or their representatives were required to complete a written informed consent agreement before the start of the assessment. Refusal did not affect the care provided to the person.

BelRAI Social Supplement

The BelRAI Social Supplement is a supplement to the BelRAI Screener and other interRAI instruments that gathers information on the social context of community-dwelling adults with care needs. The instrument and its manual have known several iterations as this study is part of the final stage of a larger evidence-based policy research project using the revised Medical Research Council (MRC) framework for the design, evaluation and implementation of complex interventions (Craig et al., 2013; Van Doren et al., 2022). The version of the BelRAI Social Supplement that was used in this study has skip patterns² and consists of a maximum of 80 items that are categorized into four sections [for more information on these sections, see Van Doren et al. (2021a)]:

A. Environmental assessment;

B. Civic engagement;

C. Psychosocial well-being; and

D. Informal care and support.

Statistical analysis

We compared individual items from the BelRAI Social Supplement between the two assessments using the proportion of observed agreement and kappa coefficients. Proportion of observed agreement (Po) is the ratio of exact agreement between the assessors as a function of the total number of assessments. The Kappa coefficient is a measure that represents the agreement between two assessors, corrected for chance. For nominal items, unweighted Cohen’s kappa (κ) was used. For ordinal items, weighted quadratic Cohen’s kappa (κw) was used. Kappa coefficient values—and thus, the strength of the association—were considered to be poor when below 0.40, moderate when ranging between 0.41 and 0.60, substantial from 0.61 to 0.80, and almost perfect from 0.81, in line with Landis and Koch (1977).

For dummy variables or binary items, bias and prevalence effects may result in skewed agreement parameters (Byrt et al., 1993; Hoehler, 2000; Vach, 2005). As highlighted by Wellens et al. (2012c), the prevalence of item scores affects the stability of the kappa coefficients. If the assessments of the sample of individuals are homogeneous (and thus, lack variability), it is unlikely that κ will approximate the maximum score of 1. This is independent of sample size. To tackle this, we calculated the prevalence index (Sim and Wright, 2005): the absolute value of the difference between the number of cases rated as positive by both raters, and the number of cases rated as negative by both raters; divided by the total number of assessments. Bias represents the degree to which the assessors disagree on the share of cases in a category. It affects the interpretation of the magnitude of κ. When bias is sizeable, kappa tends to be higher than when bias is low or absent (Feinstein and Cicchetti, 1990; Byrt et al., 1993). It is calculated by taking the absolute value of the difference between the number of cases rated as positive by rater A and negative by rater B, and the number of cases rated as negative by rater A and positive by rater B; divided by the total number of assessments.

In addition to the weighted kappa coefficient, intraclass correlation coefficients (ICC, two-way random model, absolute agreement) and 95% confidence intervals (95% CI) were calculated for ordinal items (Fleiss and Cohen, 1973; Laschinger, 1992; Field, 2005). All tests were conducted in SPSS Version 26.

Results

Between August and November 2021, nine assessors participated and a total of 52 individuals with care needs at home were included in the study. The assessors were trained social workers active at Flemish social care services organizations.

The mean age of participating adults with care needs was 72 years (range: 35–92), and 71% was female. The majority of our sample (67.3%) was over 65. 36.5% of the sample owned their place of living with no current loan or mortgage, 34.6% rented from a social housing association, and 19.2% rented on the private housing market. The remaining 9.7% either owned with a current mortgage or loan or had an alternate living arrangement (e.g., cohabitating with family members for free).

Demographic information about the individuals with care needs and the assessors can be found in Table 1.

TABLE 1

Table 1. Demographic information on community dwelling adults with care needs (n = 52) and assessors (n = 9).

The agreement statistics in Table 2 show that the level of agreement between assessors was generally high. When calculating the kappa mean (0.74) and median (0.79) values for nominal items, we observed substantial agreement. The kappa mean and median values for ordinal items were 0.81 and 0.90, which both correspond to almost perfect agreement. Following the traditional cut-off points for the interpretation of the kappa statistic, reliability was almost perfect (κ > 0.81) for 49% of all items, substantial (0.60 < κ ≤ 0.80) for 33%, moderate (0.40 < κ ≤ 0.60) for 8%, and poor (κ < 0.40) for 10%.

TABLE 2

Table 2. Agreement statistics for each item of the BelRAI Social Supplement (n = 52).

Agreement appears to be poor for eight out of 80 items. Three of these items are related to housing conditions: assessments regarding the disrepair of the house (κ = 0.37, P_o = 0.94), squalid living conditions (κ = 0, P_o = 0.96), and limited access to the dwelling or rooms in the dwelling (κ = 0.20, P_o = 0.73). The other five items are about the person’s understanding of the Dutch language (κ = 0.24, P_o = 0.75), their ability to continue caring activities for other persons (κ = 0.22, P_o = 0.89), the loss of income due to caring activities (κ = 0.00, P_o = 0.96), the IADL help they receive (κ = 0.37, P_o = 0.94), and help with social inclusion (κ = 0.36, P_o = 0.81).

For 41 binary items, we calculated the bias and prevalence index. The bias index ranged between 0 and 0.43, the prevalence index between 0.13 and 0.96. Mean and median observed agreements were 0.89 and 0.91, respectively. For 84% of all items, the observed agreement was greater than 0.80.

Discussion

This study is part of the final stage of a larger evidence-based policy research project about developing and testing a BelRAI Social Supplement using the revised Medical Research Council (MRC) framework for the design, evaluation and implementation of complex interventions (Craig et al., 2013; Van Doren et al., 2022). Testing the interrater reliability of the instrument is a key step prior to the final evaluation and implementation of the BelRAI Social Supplement in the field as of June 2022. In the present study, two independent raters simultaneously assessed home care clients. Evaluating the reliability of the assessment and quality of its data is important for all stakeholders who use the collected data (for care planning, policymaking and/or financing). When its users start questioning the (perceived and actual) quality of the data, this “uncertainty can, in turn, create a laissez-faire attitude and an incorrect use of the assessment and output in practice” (Wellens et al., 2012a,b; Van Doren et al., 2021b, p.3).

Individual items which comprise the BelRAI Social Supplement were examined using the Cohen’s kappa statistic. Strength of kappa agreement was in general substantial to almost perfect, with high proportions of observed agreement. This indicates that there is little disparity between raters on most items and that interrater reliability is high. Agreement appears to be poor for eight out of 80 items. Six items had a poor kappa value, despite a high observed agreement [disrepair of the home (κ = 0.37, P_o = 0.94), squalid conditions (κ = 0.00, P_o = 0.96), unable to continue caring activities (κ = 0.22, P_o = 0.89), loss of income due to caring activities (κ = 0.00, P_o = 0.96), IADL help (κ = 0.37, P_o = 0.94), social inclusion (κ = 0.36, P_o = 0.81)]. This combination reflects a lack of variety in answers on the particular items (homogeneity), which impacts the kappa value (Sim and Wright, 2005; Vach, 2005). This result can be supported by these items showing high values on the prevalence index.

However, two items showed a low kappa value (κ < 0.40) in combination with a low proportion of observed agreement (P_o < 0.80) [limited access to home/rooms (κ = 0.20, P_o = 0.73) and understanding Dutch (κ_w = 0.24, P_o = 0.75)]. Despite the fact that these items initially seem easy to assess, the manual contains a large number of guidelines to correctly code both items. While the first item is a dummy, the second item uses a Likert scale with six different answer options to define a person’s ability to understand Dutch. Based on the results of the present study, we can provide suggestions for optimization of both the items and the manual, for example adding more precise guidelines for scoring and adding examples to clarify the differences between the coding categories.

Considering the frailty of the population and the strict COVID-19 guidelines in place at the time of our data-collection, we decided to use the method of simultaneous rating for all cases. Simultaneous rating has three major benefits in our study: (1) it minimizes the burden of assessment for people with care needs and their caregivers, (2) it limits the inaccuracies of people’s self-reports and rating differences due to chances in past few days (Wellens et al., 2012b), and, (3) it reduces the odds of spreading COVID-19.

In this study, we used the method of simultaneous rating instead of separate raters. This method differs from other studies on the interRAI instruments for persons living at home (Hirdes et al., 2008; Kim et al., 2015). In those studies, a social worker or registered nurse assesses the person on their regular home visit while a second assessor has to repeat the assessment within 72 h. This way, each assessor can choose how to steer the conversation and obtain all the answers to complete the instrument. However, Kim et al. (2015, p. 224) clarify that “some home care cases were assessed simultaneously by the paired independent raters because of the limited visitation schedule during the observation period for the assessment.”

Limitations

COVID-19 restrictions had a large impact on the planning and execution of the home visits for our data collection. This study has been postponed multiple times over a period of a year and half. Initially, we planned an in-person refreshment course for all participating assessors to provide the training for the new version of the BelRAI Social Supplement. Instead, we chose to organize an online refreshment course to limit personal contact and minimize the time commitment for the social workers who were already severely under pressure because of delays in their daily activities. Nevertheless, we only selected assessors who received a full training on the previous version of the BelRAI Social Supplement, attended multiple discussion groups and used the instrument with their clients for 10 assessments during a testing phase of the larger scale project (Van Doren et al., 2022). Additionally, we planned to complete 100 home visits, but due to the unpredictable nature of COVID-19 surges, we finalized the data collection in November 2021, when Belgium entered its fourth wave of infections/hospital admissions and a contact restrictions were announced (Federal Public Service of Health, Food Chain Safety and Environment, 2022).

Due to the skip pattern used in this version of the BelRAI Social Supplement, one item in section D (Informal care and support) regarding the nature of the person’s relationship with a second informal carer (“Relationship to person 2” in Table 2) had too few observations to conduct the analysis on (n = 4).

Conclusion

Based on simultaneous rating of 52 community-dwelling adults in Flanders, Belgium, we calculated the interrater reliability of the latest version of the Social Supplement to the BelRAI Screener and Home Care instrument. Based on Cohen’s kappa and intraclass correlation coefficients, we found that 82% of items had substantial to almost perfect reliability. This indicates that there is limited disparity between raters on most items and that interrater reliability is high. A small number of items had a low kappa value, which was likely to a low number of and homogeneity of responses. Overall, our findings indicate that the BelRAI Social Supplement is ready to be used in Flemish home care assessments, with the provision that some items and the corresponding manual require further fine-tuning.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Social and Societal Ethics Committee of KU Leuven (SMEC). The patients/participants provided their written informed consent to participate in this study.

Author contributions

SVD, KH, and AD developed the research design. SVD collected the data and analyzed them with DDC. SVD led the writing of the manuscript. All authors contributed to the interpretation of findings, reviewed multiple versions of the manuscript, and approved the submitted version.

Funding

Financial support was obtained from the Flemish Policy Research Center for Welfare, Public Health and the Family with grant number: ZKD2376.

Acknowledgments

We are grateful to all participants of Stage I, II, and III of this study.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer RW declared a shared affiliation with the author(s) to the handling editor.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^ SVD was the main researcher in this study. She has multiple years of experience with using the different versions of the BelRAI Social Supplement.
2. ^ Skip patterns or branching questions are used to alter the sequential flow of a survey or instrument based on the answers to prior questions, or other known information about the respondent. This allows respondents to skip questions or sections not relevant for them (IPSOS, 2016).

References

Berg, K., Finne-Soveri, H., Gray, L., Henrard, J. C., Hirdes, J., Ikegami, N., et al. (2009). Relationship between interRAI HC and the ICF: Opportunity for operationalizing the ICF. BMC Health Serv. Res. 9:47. doi: 10.1186/1472-6963-9-47

PubMed Abstract | CrossRef Full Text | Google Scholar

Borrell-Carrió, F., Suchman, A. L., and Epstein, R. M. (2004). The biopsychosocial model 25 years later: principles, practice, and scientific inquiry. Ann. Fam. Med. 2, 576–582. doi: 10.1370/afm.245

PubMed Abstract | CrossRef Full Text | Google Scholar

Boscart, V., Taucar, L. S., Heyer, M., Kellendonk, T., Johnson, K., Davey, M., et al. (2021). InterRAI acute care instrument for seniors in Canadian hospitals: findings of an inter-rater reliability pilot study. Can. J. Nurs. Res. 53, 155–161. doi: 10.1177/0844562120920513

PubMed Abstract | CrossRef Full Text | Google Scholar

Byrt, T., Bishop, J., and Carlin, J. B. (1993). Bias, prevalence and kappa. J. Clin. Epidemiol. 46, 423–429. doi: 10.1016/0895-4356(93)90018-V

CrossRef Full Text | Google Scholar

Carpenter, I., and Hirdes, J. (2013). “Using interRAI assessment systems to measure and maintain quality of long-term care”, in A Good Life in Old Age: Monitoring and Improving Quality in Long-term Care (Paris: OECD Publishing) doi: 10.1787/9789264194564-7-en

CrossRef Full Text | Google Scholar

Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., and Petticrew, M. (2013). Developing and evaluating complex interventions: the new medical research council guidance. Int. J. Nurs. Stud. 50, 587–592. doi: 10.1016/j.ijnurstu.2012.09.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Declercq, A., Gosset, C., Paepen, B., De Almeida Mello, J., Vanneste, D., Detroyer, E., et al. (2009). Actieproject BelRAI III: Haalbaarheid van de RAI-methode in België. Brussel, Belgie: FOD Volksgezondheid, Veiligheid van de Voedselketen en Leefmilieu.

Google Scholar

Engel, G. L. (1980). The clinical application of the biopsychosocial model. Am. J. Psychiatry 137, 535–544. doi: 10.1176/ajp.137.5.535

PubMed Abstract | CrossRef Full Text | Google Scholar

Federal Public Service of Health, Food Chain Safety and Environment, (2022). Coronavirus COVID-19 - News Brussels. Available online at: https://www.info-coronavirus.be/en/news/occ-1711/ (accessed April 15, 2022).

Google Scholar

Feinstein, A. R., and Cicchetti, D. V. (1990). High agreement but low kappa: I. the problems of two paradoxes. J. Clin. Epidemiol. 43, 543–549. doi: 10.1016/0895-4356(90)90158-L

CrossRef Full Text | Google Scholar

Field, A. P. (2005). “Intraclass correlation,” in The Encyclopedia of Statistics in Behavioral Science, eds B. S. Everitt and D. C. Howell (New York: Wiley), 948–954.

Google Scholar

Fleiss, J. L., and Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ. Psychol. Meas. 33, 613–619. doi: 10.1177/001316447303300309

CrossRef Full Text | Google Scholar

Fried, L. P., Ferrucci, L., Darer, J., Williamson, J. D., and Anderson, G. (2004). Untangling the concepts of disability, frailty, and comorbidity: Implications for improved targeting and care. J. Gerontol. A Biol. Sci. Med. Sci. 59, M255–M263. doi: 10.1093/gerona/59.3.M255

PubMed Abstract | CrossRef Full Text | Google Scholar

Gray, L. C., Berg, K., Fries, B. E., Henrard, J.-C., Hirdes, J. P., Steel, K., et al. (2009). Sharing clinical information across care settings: The birth of an integrated assessment system. BMC Health Serv. Res. 9:71. doi: 10.1186/1472-6963-9-71

PubMed Abstract | CrossRef Full Text | Google Scholar

Gray, L. C., Bernabei, R., Berg, K., Finne-Soveri, H., Fries, B. E., Hirdes, J. P., et al. (2008). Standardizing assessment of elderly people in acute care: the interRAI acute care instrument. J. Am. Geriatr. Soc. 56, 536–541. doi: 10.1111/j.1532-5415.2007.01590.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters: Advanced Analytics, LLC. Gaithersburg, MD: Advanced Analytics LLC.

Google Scholar

Heckman, G., and Jónsson, P. V. (2018). “Comprehensive geriatric assessment: The specific assessment technology of InterRAI,” in Oxford Textbook of Geriatric Medicine, 3 Edn, eds J.-P. Michel, B. L. Beattie, F. C. Martin, and J. D. Walston (Oxford: Oxford University Press), 127–134. doi: 10.1097/PTS.0000000000000462

PubMed Abstract | CrossRef Full Text | Google Scholar

Hirdes, J. P., Ljunggren, G., Morris, J. N., Frijters, D. H., Soveri, H. F., Gray, L., et al. (2008). Reliability of the interRAI suite of assessment instruments: A 12-country study of an integrated health information system. BMC Health Serv. Res. 8:277. doi: 10.1186/1472-6963-8-277

PubMed Abstract | CrossRef Full Text | Google Scholar

Hirdes, J. P., van Everdingen, C., Ferris, J., Franco-Martin, M., Fries, B. E., Heikkilä, J., et al. (2020). The interRAI suite of mental health assessment instruments: An integrated system for the continuum of care. Front. Psychiatry 10:926. doi: 10.3389/fpsyt.2019.00926

PubMed Abstract | CrossRef Full Text | Google Scholar

Hoehler, F. K. (2000). Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. J. Clin. Epidemiol. 53, 499–503. doi: 10.1016/S0895-4356(99)00174-2

CrossRef Full Text | Google Scholar

Hood, R., Goldacre, A., Abbott, S., and Jones, R. (2022). Patterns of demand and provision in English adult social care services. Br. J. Soc. Work. [Epub ahead of print]. doi: 10.1093/bjsw/bcac011

CrossRef Full Text | Google Scholar

IPSOS (2016). Ipsos Encyclopedia - Skip Pattern (Aka Branching Question). Available online at: https://www.ipsos.com/en/ipsos-encyclopedia-skip-pattern-aka-branching-question#::̃text=A%20Skip%20Pattern%2FBranching%20Question,questions%2Fsections%20in%20the%20survey (accessed April 27, 2022).

Google Scholar

Kim, H., Jung, Y. I., Sung, M., Lee, J. Y., Yoon, J. Y., and Yoon, J. L. (2015). Reliability of the interRAI long term care facilities (LTCF) and interRAI home care (HC). Geriatr. Gerontol. Int. 15, 220–228. doi: 10.1111/ggi.12330

PubMed Abstract | CrossRef Full Text | Google Scholar

Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Int. J. Nurs. Stud. 48, 661–671. doi: 10.1016/j.ijnurstu.2011.01.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Kraus de Camargo, O. (2011). Systems of care: Transition from the bio-psycho-social perspective of the international classification of functioning, disability and health. Child Care Health Dev. 37, 792–799. doi: 10.1111/j.1365-2214.2011.01323.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Landis, J. R., and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, 159–174. doi: 10.2307/2529310

CrossRef Full Text | Google Scholar

Laschinger, H. K. (1992). Intraclass correlations as estimates of interrater reliability in nursing research. West J. Nurs. Res. 14, 246–251. doi: 10.1177/019394599201400213

PubMed Abstract | CrossRef Full Text | Google Scholar

Martin, L., Hirdes, J. P., Fries, B. E., and Smith, T. F. (2007). Development and psychometric properties of an assessment for persons with intellectual disability—the interRAI ID. J. Policy Pract. Int. Dis. 4, 23–29. doi: 10.1111/j.1741-1130.2006.00094.x

CrossRef Full Text | Google Scholar

North, J. (2020). Achieving Person-Centred Health Systems: Evidence, Strategies And Challenges. Cambridge: Cambridge University Press.

Google Scholar

Sim, J., and Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Phys. Ther. 85, 257–268. doi: 10.1093/ptj/85.3.257

CrossRef Full Text | Google Scholar

Tarricone, R., and Tsouros, A. D. (2008). Home Care in Europe: The Solid Facts. Geneva: World Health Organization.

Google Scholar

Vach, W. (2005). The dependence of Cohen’s kappa on the prevalence does not matter. J. Clin. Epidemiol. 58, 655–661. doi: 10.1016/j.jclinepi.2004.02.021

PubMed Abstract | CrossRef Full Text | Google Scholar

Valderas, J. M., Starfield, B., Sibbald, B., Salisbury, C., and Roland, M. (2009). Defining comorbidity: Implications for understanding health and health services. Ann. Fam. Med. 7, 357–363. doi: 10.1370/afm.983

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Doren, S., De Coninck, D., Hermans, K., and Declercq, A. (2021a). Examining care assessment scores of community-dwelling adults in Flanders, Belgium: The role of socio-psychological and assessor-related factors. Int. J. Environ. Res. Public Health 18:11845. doi: 10.3390/ijerph182211845

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Doren, S., Hermans, K., and Declercq, A. (2021b). Towards a standardized approach of assessing social context of persons receiving home care in Flanders, Belgium: The development and test of a social supplement to the interRAI instruments. BMC Health Serv. Res. 21:487. doi: 10.1186/s12913-021-06453-w

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Doren, S., Hermans, K., and Declercq, A. (2022). Conceptualizing relevant social context indicators for people receiving home care: a multi-method approach in Flanders, Belgium. Health Soc. Care Commu. [Epub ahead of print]. doi: 10.1111/hsc.13532

PubMed Abstract | CrossRef Full Text | Google Scholar

van Dulmen, S. A., Lukersmith, S., Muxlow, J., Santa Mina, E., Nijhuis-van der Sanden, M. W., van der Wees, P. J., et al. (2015). Supporting a person-centred approach in clinical guidelines. A position paper of the allied health community–guidelines international network (G-I-N). Health Expect. 18, 1543–1558. doi: 10.1111/hex.12144

PubMed Abstract | CrossRef Full Text | Google Scholar

Vandeurzen, J. (2018). Ontwerp Van Decreet Houdende de Vlaamse Sociale Bescherming. Brussel: Vlaams Parlement.

Google Scholar

Vermeire, T. (2020). BelRAI in Vlaanderen. Brussels: Agentschap Zorg en Gezondheid.

Google Scholar

Vermeulen, B., Van Eenoo, L., Vanneste, D., and Declercq, A. (2015). Naar een Getrapt Gebruik Van De BelRAI Met De BelRAI Screener. Leuven: LUCAS.

Google Scholar

Vernimmen, B., Taelman, M., and Brusselaers, W. (2018). Pilootproject BelRAI Screener - Eindrapport. Leuven: Steunpunt Welzijn, Volksgezondheid en Gezin.

Google Scholar

Wellens, N. I., Milisen, K., Flamaing, J., and Moons, P. (2012a). Methods to assess the reliability of the interRAI acute care: A framework to guide clinimetric testing. part II. J. Eval. Clin. Pract. 18, 822–827. doi: 10.1111/j.1365-2753.2011.01685.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Wellens, N. I., Milisen, K., Flamaing, J., and Moons, P. (2012b). Methods to assess the validity of the interRAI acute care: A framework to guide clinimetric testing. J. Eval. Clin. Pract. 18, 296–306. doi: 10.1111/j.1365-2753.2010.01571.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Wellens, N. I., Van Lancker, A., Flamaing, J., Gray, L., Moons, P., Verbeke, G., et al. (2012c). Interrater reliability of the interRAI acute care (interRAI AC). Arch. Gerontol. Geriatr. 55, 165–172. doi: 10.1016/j.archger.2011.07.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Whiteneck, G. G., Harrison-Felix, C. L., Mellick, D. C., Brooks, C., Charlifue, S. B., and Gerhart, K. A. (2004). Quantifying environmental factors: A measure of physical, attitudinal, service, productivity, and policy barriers. Arch. Phys. Med. Rehabil. 85, 1324–1335. doi: 10.1016/j.apmr.2003.09.027

PubMed Abstract | CrossRef Full Text | Google Scholar

World Health Organization [WHO] (2015). WHO Global Strategy On People-Centred And Integrated Health Services: Interim Report. Geneva: World Health Organization.

Google Scholar

World Health Organization [WHO] (2019). Regional Office for Europe. Creating 21st Century Primary Care in Flanders and Beyond. Copenhagen: World Health Organization.

Google Scholar