Improved quality of physiotherapy care in patients with Whiplash-Associated Disorders: Results based on 16 years of routinely collected data

Quality improvement is now a central tenet in physiotherapy care, and quality indicators (QIs), as measurable elements of care, have been applied to analyze and evaluate the quality of physiotherapy care over the past two decades. QIs, based on Donabedian's model of quality of care, provide a foundation for measuring (improvements in) quality of physiotherapy care, providing insight into the many remaining evidentiary gaps concerning diagnostics, prognostics and treatment, as well as patient-related outcome measures. In this overview we provide a synthesis of four recently published articles from our project group on the topic of quantitative measures of quality improvement in physiotherapy care, in this context specifically focused on patients with WAD in primary care physiotherapy. A set of process and outcome QIs (n = 28) was developed for patients with WAD and linked to a database consisting of routinely collected data (RCD) on patients with WAD collected over a 16-year period. The QIs were then embedded per step of the clinical reasoning process: (a) administration (n = 2); (b) history taking (n = 7); (c) objectives of examination (n = 1); (d) clinical examination (n = 5); (e) analysis and conclusion (n = 1); (f) treatment plan (n = 3); (g) treatment (n = 2); (h) evaluation (n = 5); and (i) discharge (n = 2). QIs were expressed as percentages, allowing target performance levels to be defined ≥70% or ≤30%, depending on whether the desired performance required an initially high or low QI score. Using RCD data on primary care patients with WAD (N = 810) and a set of QIs, we found that the quality of physiotherapy care has improved substantially over a 16-year period. This conclusion was based on QIs meeting predetermined performance targets of ≥70% or ≤30%. Twenty-three indicators met the target criterium of ≥70% and three indicators ≤30%. Our recommended set of QIs, embedded in a clinical reasoning process for patients with WAD, can now be used as a basis for the development of a validated QI set that effectively measures quality (improvement) of primary care physiotherapy in patients with WAD.

Introduction

Quality improvement is no longer the preserve of a few enthusiastic professionals but has become a central tenet in healthcare, including physiotherapy. Quality improvement is now part of the daily routine of all those involved in delivering healthcare and is even a statutory obligation in many countries (1), including the Netherlands (2). Physiotherapists are directly or indirectly involved in optimal physiotherapy care not only in hospitals but also increasingly in primary care. Measurement of quality using quality indicators (QIs) plays an important role in improvement of healthcare (3–6).

Clinical registries, including routinely collected data (RCD), are recognized as an important source of data and harbor the potential to improve quality of care (7). They provide data about variation in quality of care, whether benchmarks are being met and facilitate feedback to clinicians, managers, funders, policymakers and researchers. Using clinical registries to inform data-driven quality improvement projects has resulted in the promotion of best practice and further stimulated use of registry data for quality improvement (7). Physiotherapists have monitored quality of care since the 1990's. During workshops in 1992, in which the methodology of indicator development for physiotherapy was explored, the Australian Physiotherapy Association adopted the concept of QIs to measure the quality of physiotherapy care (8). Around the same period (1990), the project “Quality in Physiotherapy” was launched in the Netherlands and resulted in the first clinical practice guideline, “Patient Documentation,” from the Royal Dutch Society for Physical Therapy (KNGF) (9). Since then, similar quality reporting programs have been implemented in the United States, Canada, Australia and Europe, and a number of books and articles have been published that address various aspects of the quality of care in general (3–6) and Dutch physiotherapy in particular (10–19). However, despite the increasing availability over the past decade of QIs designed to manage a variety of rheumatic and musculoskeletal diseases, the use of QIs in physiotherapy is still limited (20). Anno 2022, quality of physiotherapy still remains an important topic across various physiotherapy domains, including the domain Whiplash-Associated Disorders (WAD).

A particularly complex domain within physiotherapy is the quality of care in patients with WAD, a condition that is often presented to physiotherapists and remains difficult to manage. Whiplash injury is one of the most common traffic-related injuries (21) and is caused by acceleration-deceleration forces acting on the neck, head and torso (22, 23). The impact may result in lesions of cervical spine structures and effects on sensory, motor and mental functions, which in turn can lead to a variety of clinical manifestations, including neck pain, neck stiffness, headache, dizziness, tinnitus, paresthesia, loss of balance, loss of eye movement control, cognitive manifestations, and pain sensory disturbances indicative of sensitization of the peripheral and central nervous systems (24–30). These clinical manifestations are classified as WAD (31, 32). Worldwide, physiotherapy is one of the preferred treatment options for patients with WAD, especially when combined with other treatments such as medication (33). International data indicate that approximately 50% of people who sustain a whiplash injury will not recover and will continue to experience ongoing disability and pain 1 year after injury (34, 35). In addition to the poor prospects for recovery, poor treatment responses are another important issue (32, 36, 37). Today, many evidentiary gaps remain in terms of diagnostics, prognostics, and treatment, as well as concerning patient-related outcome measurements in patients with WAD (33, 38).

Clinical practice guidelines (CPGs) enable physiotherapists to assess (1) the extent to which physiotherapy management and assessment aligns with available research-based evidence, and (2) gaps in practice that need improvement. Routinely collected data (RCD) describing real practice populations, such as patients with WAD, can fill these evidentiary gaps and act an important driver of quality improvement and performance-based measurement (7, 39).

The present paper is an overview and synthesis of four recently published articles from our project group concerning the development and application of QIs in physiotherapy primary care. Using a routinely collected dataset, these papers explored quantitative measures of quality improvement in physiotherapy care in patients with WAD, based on the development and application of QIs embedded in the clinical reasoning process (40–43). Summarizing these papers, we introduce readers to the specific methodology of developing and applying QIs in patients with WAD in physiotherapy primary care. This approach can provide a framework and state of the art example for future QI research initiatives involving topics such as comparability of practitioners, inter-rater reliability, sensitivity to change and predictive validity.

Clinical practice guidelines

Current national and international Clinical Practice Guidelines (CPGs) for patients with WAD are mainly based on systematic reviews and on primary studies of diagnostics, prognostics, and treatment outcomes (44–52). In general, comparable recommendations can be found across these guidelines, all of which are based on weak or moderate levels of evidence.

The Dutch CPG “Physiotherapy Management and WAD” was introduced in 2001 (44) and updated in 2002 (45) and 2016 (46). The content of Dutch CPGs is organized in accordance with a nine-step clinical reasoning process, in combination with the best available evidence and professional consensus. The clinical reasoning cycle is an internationally accepted concept to facilitate problem solving and decision making in the daily practice of physiotherapy. The transparency of this clinical reasoning process is considered a cornerstone of the quality of physiotherapy care (53).

Data are lacking on the complexity of the clinical reasoning process in patients with WAD (54). The lack of a detailed understanding of the clinical reasoning process related to various features of WAD may hamper the implementation of WAD-related CPGs in clinical practice and the delay improvement of physiotherapy quality in primary care.

Defining quality indicators in healthcare using Donabedian's model

QIs have been defined as “measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality of the care provided” (4, 5). They do not measure quality directly but are auxiliary variables that indirectly reflect the quality of care through ratios, thus one could also speak of quality-related indicators. Most initiatives to evaluate (improvement of) quality of care are consistent with Donabedian's model (55). Donabedian argued that the evaluation of context, process, outcome and structure indicators and their mutual relationships all provide a comparable picture of quality of care in different settings (55–57). Donabedian then postulated “relationships between the constructs of structure, process and outcome, based on the assumption that good structure should promote good processes and good processes should in turn promote good outcomes in a reciprocal pathway.”

“Structure” is defined as the professional and organizational resources associated with the provision of healthcare (e.g., availability of physiotherapy, equipment and staff training), “process” as the things done to and for the patient (e.g., practice referrals, clinical reasoning and decision), and “outcome” as the desired result of care provided by the health practitioner (e.g., a patient's functioning, and satisfaction with quality of care) (55–57). Context indicators were added to the postulated relationships and are indicators “that together constitute the complete context of an individual's life and living, and the background of an individual's health and health-related states in particular” (58).

Development of quality indicators

The most commonly used method for the development of QIs is an iterated consensus rating procedure (5, 59, 60), such as the systematic RAND-modified Delphi method. By including independent expert comments and iterative feedback, this method results in a set of recommendations with good face validity and suitable for transcription into QIs (61).

The preferred method of QI development consists of five steps: (a) extraction of recommendations from CPGs, patient-related outcome measurements, and literature, particularly systematic reviews; (b) transformation of recommendations into QIs by phrasing them as the average degree (in %) to which patients were subjected to a methodically performed clinical reasoning process; (c) appraisal by an expert and user panel, including scoring of the set of QIs on a five-point Likert scale (1 = not at all to 5 = completely) based on acceptability, feasibility, clarity, and relevancy to the physiotherapy care process; (d) classification of process indicators into the nine steps of the clinical reasoning process; and (e) classification of outcome indicators in accordance with the International Classification of Functioning, Disability and Health (ICF) (58), e.g., body functions, activity and participation, as well as personal and environmental factors.

The methods used for indicator development in physiotherapy will be briefly explained by means of a recently published example concerning the quality of physiotherapy care in patients with WAD (40, 43). Two specialized physiotherapists independently extracted recommendations related to the nine steps of the physiotherapy clinical reasoning process, using sources including the Dutch CPG Physiotherapy Management and WAD (44, 45), the Quebec Task Force on WAD (31) and the updated Dutch CPG Neck Pain (including WAD) (46). Both physiotherapists were involved in the development of these CPGs. Following critical evaluation and checking for duplicates or overlap, 125 preselected items could be reduced to 96 and compared to current evidence (33, 38). Phrasing them as the average degree (in %) to which patients were subjected to a methodically performed clinical reasoning process, the 96 items were then transformed into a set of 28 QIs.

In the set of guideline-based QIs, quantified as percentages ranging from 0 to 100%, the number of times a QI was met was designated as numerator and the total number of patients was designated as denominator, thus N = 810 unless stated otherwise. We give some examples from the available WAD patient dataset in Table 2. For example, the numerator score for the number of patients subjected to previous medical imaging neck diagnostics (noted as yes) was 178/810 (QI = 21.9%); the extent to which physical examination objectives were formulated in agreement with patients' history taking (noted as yes) was 810/810 (QI = 100%), and the extent to which treatment goals were in agreement with the prognostic health profile and time phase since accident (noted as yes) was 529/810 (QI = 65.3%).

The level of research evidence for the formulated QIs, from levels I to IV, was determined based on a national consensus document (62), “with level I being the highest: level I = systematic review or >2 high-quality controlled trials or high-quality diagnostic studies or high-quality psychometric studies; level II = two high-quality controlled trials or high-quality diagnostic studies or high-quality psychometric studies; level III = high-quality non-controlled trials or low-quality diagnostic studies or low-quality psychometric studies; level IV = expert opinion and professional consensus or standard.” The level of evidence for most QIs was based on professional consensus (level IV) (40). Many reviews have called for further research to identify who does or does not respond to treatment. To date, clinical trials of WAD have not been able to identify factors associated with treatment response. Sterling et al. stated: “It would be fair to say that for musculoskeletal conditions, including neck pain and WAD, little progress has been made in this direction” (33).

Set of quality indicators in steps of clinical reasoning in patients with WAD

Clinical reasoning has been defined “as a process in which the physiotherapist, interacting with the patient and significant others, structures meaning, goals and health management strategies based on scientific evidence, clinical data, client choices and professional judgment and knowledge” (53, 63, 64). The set of QIs are embedded in the nine steps of clinical reasoning, with the number of QIs assigned to each step indicated in parentheses: I: Patients' information (n = 2), II: History taking, (n = 7) III: Objectives of examination (n = 1), IV: Clinical examination (n = 5), V: Analysis and conclusion (n = 5), VI: Treatment plan (n = 3), VII: Treatment (n = 2), VIII: Evaluation (n = 5), and IX: Discharge (n = 2) (30–33).

Table 1 presents an overview of the complete set of QIs (n = 28) for the physiotherapy clinical reasoning process in patients with WAD (40–43) and also includes the items and the level of evidence per indicator.

TABLE 1

Table 1. Quality indicators for physiotherapy care process in patients with Whiplash-Associated Disorders (WAD): steps of clinical reasoning, number of indicators, type of indicator, item measured, indicator, and level of evidence^* [adapted from Oostendorp et al. (40, 43)].

Routinely collected dataset of patients with WAD

The first WAD pen and paper patient record was introduced in 1996 in two primary care physiotherapy practices based on the first CPG Patient Documentation (9). The Medical Ethics Committee of Radboud University Medical Center Nijmegen, the Netherlands, waived in writing the requirement for ethical approval as the dataset involved routinely collected data that represented no extra burden for participating patients (www.ru.nl/rdm/collecting-data/informed-consent-ethics-committees).

The participating physiotherapists received updates in accordance with adjustments to the content of the most recent CPG and the adapted patient record files, explained in 3 h meetings in 2001, 2002, 2009, and 2016, respectively. They also received instructions on how to score items for each step of the clinical reasoning process. All patient records were archived and relevant characteristics of the dataset are presented below and in Table 2, ordered according to the diagnostic, therapeutic and evaluative steps of clinical reasoning.

TABLE 2

Table 2. Item scores per indicator of diagnostic clinical reasoning process in patients with Whiplash-Associated Disorders (WAD) [adapted from Oostendorp et al. (40)] N = 810; n (%) unless otherwise stated.

Sociodemographic characteristics of the total group (N = 810) are presented in Table 2. Patient's mean age was 43.0 years (SD 12.6) and 586 (72.3%) were female. The most frequent accident-related characteristics were direction of impact (back n = 512; 63.2%), neck trauma without head trauma (n = 572; 70.6%), and onset of whiplash-related complaints within 3 days (n = 556; 68.9%).

Diagnostic steps of the clinical reasoning process

An overview of the item scores per step of the diagnostic part of the clinical reasoning process is presented in Table 2, adapted from Oostendorp et al. (40).

Most patients (n = 555; 68.5%) were classified as WAD-2, with a delayed recovery (n = 441; 54.4%), and were referred 7 weeks to >6 months after the accident (n = 459; 56.7%). Eighty-one patients (10.0%) reported a previous history of neck injury, and 144 patients (17.8%) a history of neck pain and stiffness. Half of the patients had been previously treated with several interventions, such as pain medication (n = 369; 45.6%), cervical soft collar (n = 514; 63.4%) or (manual) physiotherapy (n = 332; 40.0%). No patients were fully recovered and 43 (5.3%) were partially recovered. The results of earlier treatment were inestimable in 190 patients (23.5%), while in 263 patients (32.5%) a stabilization in functioning was estimated, and 314 patients (38.8%) showed a deterioration in functioning.

A number of potentially negative prognostic factors for recovery were found, including pain intensity (high level of pain intensity in the acute phase), low level of functioning, recovery rate since accident (inestimable [n = 369; 45.6%] and delayed recovery [n = 441; 54.4%]), modified Waddell's non-organic physical signs (>3: n = 530; 92.2%), risk for passive coping (n = 443; 54.7%) and risk for fear avoidance (n = 467; 57.7%). The prognostic factors were summarized in three prognostic recovery profiles, ranging from a positive profile (profile A) to a negative profile (profile C). Profile B is characterized by both positive and negative factors, making it difficult to estimate (inestimable) the chance of recovery (Profile B). No patients were classified in prognostic health profile A, 369 (45.6%) in profile B and 441 patients (54.4%) in profile C.

In conclusion, all patients developed persistent symptoms ranging from mild to severe pain and disability following their accident. They were referred more than 3 months after their accident (most recovery occurs within the first 3 months after which time the condition tends to plateau), and a majority of patients used a cervical collar (brace) during 4 weeks on average. Around half of the patients showed a delayed recovery rate following their accident, and the remaining group had an inestimable recovery time.

Based on clinical analysis and consequent conclusions, and following consultation with the patient and the patient's referring physician concerning an indication for physiotherapy, physiotherapy was possibly indicated in 178 patients (22.0%) and definitely indicated in 632 patients (78.0%), classified in phases from 1 to 6.

Therapeutic and evaluative steps of the clinical reasoning process

An overview of the item scores per step of the therapeutic and evaluative part of the clinical reasoning process for the total group (N = 810) is presented in Table 3, adapted from Oostendorp et al. (40, 41).

TABLE 3

Table 3. Item scores of therapeutic and evaluative process of clinical reasoning process in patients with Whiplash-Associated Disorders (WAD) [adapted from Oostendorp et al. (40)] N = 810; n (%) unless otherwise stated.

The settings of treatment goals were in agreement with the prognostic health profiles and the time phases 1–6 since the accident in 529 (65.3%) of 810 patients but in disagreement in 281 patients (34.7%). Physiotherapy modalities were in agreement with treatment goals and best available evidence in 366 (69.2%) of 529 patients but in disagreement in 163 patients (30.8%). The pre-estimated treatment duration was >6 months in 516 patients (63.7%) and the pre-estimated number of treatment sessions was ≥16 in 479 patients (59.1%). Patient-related outcome measurements were available in 523 patients. Intensity of pain was reduced to ≤30 (Visual Analogue Scale [VAS] 0–100) in 301 patients (59.3%) and functioning was improved to ≤14 (Neck Disability Index [NDI] 0–50) in 191 patients (36.5%). Approximately half of the patients (n = 241; 46.1%) were improved based on the global perceived improvement scale (GPE from “improved” [very good, good and fairly improved], to “no change” [same as before] and “worse” [worse and much worse]).

The treatment plan for about two thirds of patients was in line with the time phase after accident and the prognostic profile. However, this was not the case in around one third of patients. Therefore, if patients were assigned the correct time phases and prognostic profiles, the composition of treatment modalities suits the treatment plan in more than two thirds of cases, but is discordant with the treatment plan in around one third of patients. In conclusion, we can therefore safely conclude that there is abundant room for data-driven quality improvement of physiotherapy management in patients with WAD.

Despite the poor prospects for functional recovery at initial contact with the physiotherapist, about half of the patients rated the perceived treatment effect as “improved,” ranging from very improved to fairly improved, and more than half of all patients rated a reduction in the intensity of pain (to minimal pain), while in around one third of patients functioning was improved to “optimal functioning.” These patient-related outcomes underline the fact that around 50% of patients were not recovered at 1 year and experience ongoing disability and pain after a whiplash-related injury.

In contrast to longitudinal studies (34, 65–70), the data presented here only include data gathered during the treatment episode, without additional follow-up. Within these limitations, about half of the patients improved while the other half were categorized as “no change” or “worse,” without meaningful differences related to the year of treatment or the phase after whiplash-related injury. Based on the results of longitudinal studies of functional recovery after whiplash-related injury, it seems unlikely that recovery rates of the described patients will improve substantially in the future. International data also indicate that ~50% of people involved in a whiplash-causing accident will not recover and will continue to experience ongoing activity limitations, participation problems, and long-term neck pain (34, 35).

From dataset to quality indicator percentages

The formula for percentages of each QI is used as a sum score of the percentages of the dichotomized items, divided by the number of items per indicator. The overall QI scores are used as unweighted sum scores of the percentages per year as numerator and the number of years (n = 16) as denominator. The QIs per step of the clinical reasoning process are expressed as mean percentages (including standard deviation, minimum and maximum, and median).

To facilitate interpretation of performance targets, QI percentage scores were classified from “negligible” (0%) to “excellent” (100%) or from “excellent (0%) to “negligible” (100%), depending on the direction of the indicator (0–20% “negligible” or “excellent”; 21–40% “weak” or “good”; 41–60% “sufficient”; 61–80% “good” or “weak”; 81–100% “excellent or ‘negligible'). For instance, the direction of inventory of prognostics factors” (Indicator 9) was from 0% (negligible) to 100% (excellent) and the direction of previous diagnostics (Indicator 7) from 0% (excellent) to 100% (negligible). Most indicators were categorized from 100 to 0% (from excellent to negligible), and three indicators went from 100 to 0% (from negligible to excellent) (40).

A desired performance target regarding quality of physiotherapy care can be determined in consultation with different stakeholders. In consultation with physiotherapists working in primary care. the Dutch Royal Association for Physical Therapy (KNGF) has established target standards for QIs related to steps of the clinical reasoning process (≥70% and ≤30% depending on the desired direction of the indicator. In the current context, the performance target was set to ≥70% for 23 indicators (QIs 1–4, 8–15, 17–27) and to ≤30% for 3 indicators (QIs 5–7), while two indicators (QIs 16 and 28) remained non-defined due to their innovative character. See Table 4. The indicator “the number of patients in whom symptoms related to central sensitization are present” needs further elaboration. There is growing evidence to suggest that chronic WAD is associated with impairments in generalized sensory hypersensitivity as a result of sensitized pathways within the central nervous system (24, 25).

TABLE 4

Table 4. Long term evaluation of quality of clinical reasoning process of physiotherapy in patients with Whiplash-Associated Disorders (n = 810 unless otherwise stated).

Application of WAD-related quality indicators in the clinical reasoning process

Using the set of QIs together with our routinely collected dataset, the quality of physiotherapy care in patients with WAD was evaluated. To translate data into QI scores expressed as frequencies, we formulated algorithms that followed the process of clinical reasoning in patients with WAD, thus allowing target performance to be defined.

The percentages of QIs per step of the clinical reasoning process are presented in Table 4.

Four QIs (indicators 7, 10, 13, 20) did not meet the performance target, the target performance of two QIs (indicators 15 and 28) was non-defined as discussed above, while 22 QIs met the performance targets of ≥70% or ≤30% over a period of 16 years.

The number of positively-assessed QIs for performance targets continued to improve over a period of 16 years in which the data were collected. The most striking quality improvements were seen in the clinical examination (based on the objectives of examination), the analysis and conclusion of the diagnostic process in the transition to treatment plan and treatment, and in the frequency of use of patient-related outcome measurements such as pain intensity, functioning and global perceived effect (GPE). However, there is still room for improvement in clinical practice.

Suitability of routinely collected data for quality evaluation of physiotherapy care

RCD offer several advantages. Data collection under real-life practice conditions maximizes representativeness and generalizability, minimizes costs and effort, and allows the capture of information from large populations and many clinical practices over long periods (7, 71). However, these advantages should be viewed with caution, as errors and biases due to incomplete registration can interfere with results (7, 71, 72). Registry data are generally only visible within the local practice and are not routinely used to improve quality of physiotherapy care on a national level. To use RCD properly, certain challenges and barriers must be overcome. Reluctance of healthcare providers to supply data, poor integration in daily practice, limited availability of skills and lack of funding have been identified as the most frequent barriers to use (7).

As many evidentiary gaps persist concerning the prognostics, diagnostics and treatment of patients with WAD, the use of centralized, on-going RCD generally represents a useful alternative approach to understanding the quality of physiotherapy care. RCD on physiotherapy management in patients at different phases of WAD may provide a more complete view of the clinical reasoning process and a more comprehensive and realistic view of routine practice compared to data gathered during an RCT. In the majority of RCTs (n = 122) involving patients with non-specific neck pain the reporting of the clinical reasoning process was incomplete, specifically in the diagnostic aspect of the process, with only 6% of the RCTs including a complete diagnostic process (73).

As a counterbalance to the overreliance on RCTs as the highest level of evidence establishing treatment effectiveness, there is increasing interest in clinical research that includes a broad selection of patients, has less strict inclusion and exclusion criteria and uses patient-reported outcomes (7, 71). The number of real-life studies has been rapidly growing in different areas of medicine like respiratory medicine (74). Nevertheless, few physiotherapy studies utilizing RCD have been published to date (71). In an effort to improve assessment of the quality of real-world studies, the RECORD statement (Reporting of studies Conducted using Observational Routinely-collected health Data) was recently formulated (75). The RECORD statement is a checklist of items, including codes to identify participants and to classify patient characteristics, exposures, confounders and outcomes. Most items covered by the RECORD statement were included in our observational studies using RCD in patients with WAD (40, 42). We anticipate that the RCD underlying our observational study could plausibly act as preliminary real-world evidence concerning (manual) physiotherapy management and WAD, and could be used to improve the design of future data-driven clinical improvement studies' (76, 77).

The data on WAD patients described here were routinely collected over a period of 16 years in a large population of patients with WAD, using broad inclusion and limited exclusion criteria. These data therefore reflect the heterogeneity of real practice populations under routine care conditions, conditions that differ from the artificial situation of an RCT. To the best of our knowledge, this is the only example of the use of RCD in the (manual) physiotherapy management of WAD patients.

Concluding remarks

To our knowledge, the set of QIs discussed here is the first set to be developed specifically for measurement of the quality of physiotherapy care in patients with WAD. The good face and content validity of this set indicates suitability for application in primary care physiotherapy practice. Further research (5, 6) will be needed to provide evidence of acceptability, reliability, sensitivity to change, and predictive validity of this set of QIs of physiotherapy care in patients with WAD.

The set of QIs described here, embedded in a clinical reasoning process for patients with WAD, can be used as a starting point for research on the clinimetric properties that measure the sensitivity to change in quality of primary care physiotherapy in patients with WAD.

The combination of a variety of evidence regarding primary care physiotherapy management of patients with WAD and neck pain will provide a broader view of the clinical reasoning process, and hopefully promote a more comprehensive and realistic view of the (improvement of) quality of routine practice when compared to data gathered exclusively during an RCT (or even pragmatic clinical trials).

We would argue that routinely collected data can aid improvement of the quality of (manual) physiotherapy through benchmarking, personalization, and continued education, not only in patients with WAD, but also in other musculoskeletal (pain) conditions. Furthermore, international consensus on a set of QIs embedded in the physiotherapy clinical reasoning process, as well as on performance targets and scoring procedures, would help considerably in improving comparisons between studies of physiotherapy care quality in patients with WAD. We urge policy makers, professional Associations, Clinicians and Researchers Across the Globe to Consider Investing Resources in the development and application of QIs for monitoring and improving (physiotherapy) care for patients with WAD.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.

Author contributions

All authors contributed to drafting and revising the article, gave final approval of the version to be published, and agree to be accountable for all aspects of the work.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Salvatore FP, Fanelli S, Donelli CC, Milone D. Value-based health-care principles in health-care organizations. Int J Organ Anal. (2021) 29:1443–54. doi: 10.1108/IJOA-07-2020-2322

CrossRef Full Text | Google Scholar

2. Scholte M. General Introduction. PhD Thesis It Takes Three to Tango: Developing and Implementing Quality Indicators for Physical Therapy: Lessons Learned. Nijmegen: Radboud University Nijmegen (2017). p. 23–31.

Google Scholar

3. Lawrence M, Olesen F. Indicators of quality health care. Eur J Gen Pract. (1997) 3:103–8. doi: 10.3109/13814789709160336

CrossRef Full Text | Google Scholar

4. Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care. (2003) 15:523–30. doi: 10.1093/intqhc/mzg081

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Campbell S, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. In: Grol R, Baker R, Moss F, editors. Quality Improvement Research. Understanding the Science of Change in Health Care. London: BMJ Publishing Group (2004). p. 6–28.

6. Braspenning J, Campbell S, Grol R. Measuring changes in patient care: development and use of indicators. In: Grol R, Wensing M, Eccles M, editors. Improving Patient Care. The Implementation of Change in Clinical Practice. Edinburgh: Elsevier Butterworth Heinemann (2005). p. 222–34.

Google Scholar

7. Gawthorne J, Fasugba O, Levi C, Mcinnes E, Ferguson C, Mcneil JJ, et al. Are clinicians using routinely collected data to drive practice improvement? A cross-sectional survey. Int J Qual Health Care. (2021) 33:mzab141. doi: 10.1093/intqhc/mzab141

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Grimmer K, Dibden M. Clinical indicators for physiotherapists. Aust J Physiother. (1993) 9:81–5. doi: 10.1016/S0004-9514(14)60471-2

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Koninklijk Nederlands Genootschap Fysiotherapie. Richtlijnen voor de Fysiotherapeutische Verslaglegging (KNGF-Guidelines Physiotherapy Documentation). Amersfoort: KNGF (1993).

10. Nijkrake MJ, Keus SH, Ewalds H, Overeem S, Braspenning JCC, Oostendorp RAB et al. Quality indicators for physiotherapy in Parkinson's disease. Eur J Phys Rehabil Med. (2009) 45:239–45.

PubMed Abstract | Google Scholar

11. Jansen MJ, Hendriks EJ, Oostendorp RA, Dekker J, De Bie RA. Quality indicators indicate good adherence to the clinical practice guideline on “Osteoarthritis of the hip and knee” and few prognostic factors influence outcome indicators: a prospective cohort study. Eur J Phys Rehabil Med. (2010) 46:337–45.

PubMed Abstract | Google Scholar

12. Rutten GM, Harting J, Bartholomew LK, Schlief A, Oostendorp RA, de Vries NK. Evaluation of the theory-based Quality improvement in Physical Therapy (QUIP) programme: a one-group, pre-test post-test pilot study. BMC Health Serv Res. (2013) 13:194. doi: 10.1186/1472-6963-13-194

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Oostendorp RA, Rutten GM, Dommerholt J, Nijhuis-van der Sanden MW, Harting J. Guideline- based development and practice test of quality indicators for physiotherapy care in patients with neck pain. J Eval Clin Pract. (2013) 19:1044–53. doi: 10.1111/jep.12025

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Scholte M, Neeleman-van der Steen CW, Hendriks EJ, Nijhuis-van der Sanden MW, Braspenning J. Evaluating quality indicators for physical therapy in primary care. Int J Qual Health Care. (2014) 26:261–70. doi: 10.1093/intqhc/mzu031

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Peter WF, Hurkmans EJ, van der Wees P, Hendriks E, van Bodegom-Vos L, Vliet Vlieland TP. Healthcare quality indicators for physiotherapy management in hip and knee osteoarthritis and rheumatoid arthritis: a Delphi study. Musculoskeletal Care. (2016) 14:219–32. doi: 10.1002/msc.1133

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Gijsbers HJ, Lauret GJ, van Hofwegen A, van Dockum TA, Teijink JA, Hendriks HJ. Development of quality indicators for physiotherapy for patients with PAOD in the Netherlands: a Delphi study. Physiotherapy. (2016) 102:196–201. doi: 10.1016/j.physio.2015.06.001

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Verburg AC, van Dulmen SA, Kiers H, Nijhuis-van der Sanden MWG, van der Wees PJ. Development of a standard set of outcome measures for non-specific low bac pain in Dutch primary care physiotherapy practices: a Delphi study. Eur Spine J. (2019) 28:1550–64. doi: 10.1007/s00586-019-05962-x

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Verburg AC, van Dulmen SA, Kiers H, Nijhuis-van der Sanden MWG, van der Wees PJ. Patient-reported outcome-based Quality indicators in Dutch primary care physical therapy for patients with nonspecific low back pain: a cohort study. Phys Ther. (2021) 101:pzab118. doi: 10.1093/ptj/pzab118

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Steenbruggen RA, van Oorsouw R, Maas M, Hoogeboom TJ, Brand P, Van der Wees Ph, et al. Development of quality indicators for departments of hospital-based physiotherapy: a modified Delphi study. BMJ Open Qual. (2020) 9:e000812. doi: 10.1136/bmjoq-2019-000812

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Johansen I, Klokkerud M, Anke A, Børke JB, Glott T, Hauglie U, et al. A quality indicator set for use in rehabilitation team care of people with rheumatic and musculoskeletal diseases; development and pilot testing. BMC Health Serv Res. (2019) 19:265. doi: 10.1186/s12913-019-4091-4

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Holm LW, Carroll LJ, Cassidy JO, Hogg-Johnson S, Côté P, Guzman J, et al. The burden and determinants of neck pain in whiplash associated disorders after traffic collisions: results of the Bone and Joint Decade 2000–2010 Task Force on Neck Pain and its Associated Disorders. J Manipulative Physiol Ther. (2009) 32:S61–9. doi: 10.1016/j.jmpt.2008.11.011

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Davis CG. Mechanisms of chronic pain from whiplash injury. J Forensic Leg Med. (2013) 20:74–85. doi: 10.1016/j.jflm.2012.05.004

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Elliott JM, Noteboom JT, Flynn TW, Sterling M. Characterization of acute and chronic whiplash- associated disorders. J Orthop Sport Phys. (2009) 39:312–23. doi: 10.2519/jospt.2009.2826

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Stone AM, Vicenzino B, Lim EC, Sterling M. Measures of central hyperexcitability in chronic whiplash associated disorder–a systematic review and meta-analysis. Man Ther. (2013) 18:111–7. doi: 10.1016/j.math.2012.07.009

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Van Oosterwijck J, Nijs J, Meeus M, Paul L. Evidence for central sensitization in chronic whiplash: a systematic literature review. Eur J Pain. (2013) 17:299–312. doi: 10.1002/j.1532-2149.2012.00193.x

PubMed Abstract | CrossRef Full Text | Google Scholar

26. DePauw R, Coppieters I, Meeus M, Caeyenberghs K, Danneels L, Cagnie B. Is traumatic and non-traumatic neck pain associated with brain alterations? - A systematic review. Pain Phys. (2017) 20:245–60. doi: 10.36076/ppj.2017.260

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Daenen L, Nijs J, Raadsen B, Roussel N, Cras P, Dankaerts W. Cervical motor dysfunction and its predictive value for long-term recovery in patients with acute whiplash-associated disorders: a systematic review. J Rehabil Med. (2013) 45:113–22. doi: 10.2340/16501977-1091

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Astrup J, Gyntelberg F, Johansen AM, Lei A, Marott JL. Impaired neck motor control in chronic whiplash and tension-type headache. Acta Neurol Scand. (2021) 144:394–9. doi: 10.1111/ane.13473

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Higgins JP, Elliott JM, Parrish TB. Brain network disruption in whiplash. Am J Neuroradiol. (2020) 41:994–1000. doi: 10.3174/ajnr.A6569

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Woodhouse A, Vasseljen O. Altered motor control patterns in whiplash and chronic neck pain. BMC Musculoskelet Disord. (2008) 9:90. doi: 10.1186/1471-2474-9-90

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Spitzer WO, Skovron ML, Salmi LR, Cassidy JD, Duranceau J, Suissa S, et al. Scientific monograph of the Quebec Task Force on whiplash-associated disorders: redefining “whiplash” and its management. Spine. (1995) 20:1–73S.

PubMed Abstract

32. Sterling M. A proposed new classification system for whiplash associated disorders: implications for assessment and management. Man Ther. (2004) 9:60–70. doi: 10.1016/j.math.2004.01.006

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Sterling M, de Zoete RMJ, Coppieters I, Farrell SF. Best evidence rehabilitation for chronic pain. Part 4: neck pain. J Clin Med. (2019) 8:E1219. doi: 10.3390/jcm8081219

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Bunketorp L, Nordholm L, Carlsson J. A descriptive analysis of disorders in patients 17 years following motor vehicle accidents. Eur Spine J. (2002) 11:227–34. doi: 10.1007/s00586-002-0393-y

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Carroll LJ, Holm LW, Hogg-Johnson S, Côtè P, Cassidy JD, Haldeman S, et al. Course and prognostic factors for neck pain in whiplash-associated disorders (WAD): results of the Bone and Joint Decade 2000–2010 Task Force on Neck Pain and Its Associated Disorders. J Manipul Physiol Ther. (2009) 32:S97–−107. doi: 10.1016/j.jmpt.2008.11.014

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Michaleff ZA, Maher CG, Lin CW, Rebbeck T, Jull G, Latimer J, et al. Comprehensive physiotherapy exercise programme or advice for chronic whiplash (PROMISE): a pragmatic randomised controlled trial. Lancet. (2014) 384:133–41. doi: 10.1016/S0140-6736(14)60457-8

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Wiangkham T, Duda J, Haque S, Madi M, Rushton A, Eldabe S. The effectiveness of conservative management for acute Whiplash Associated Disorder (WAD) II: a systematic review and meta-analysis of randomised controlled trials. PLoS ONE. (2015) 10:e0133415. doi: 10.1371/journal.pone.0133415

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Damgaard Damgaard P, Bartels EM, Ris I, Christensen R, Juul-Kristensen B. Evidence of physiotherapy interventions for patients with chronic neck pain: a systematic review of randomised controlled trials. ISRN Pain. (2013) 2013:567175. doi: 10.1155/2013/567175

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Wilcox N, McNeil JJ. Clinical quality registries have the potential to drive improvements in the appropriateness of care. Med J Aust. (2016) 205:S21–6. doi: 10.5694/mja15.00921

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Oostendorp RAB, Elvers H, van Trijffel E, Rutten GM, Scholten-Peeters GG, Heijmans M, et al. Has the quality of physiotherapy care in patients with Whiplash-associated disorders (WAD) improved over time? A retrospective study using routinely collected data and quality indicators. Patient Prefer Adher. (2018) 12:2291–308. doi: 10.2147/PPA.S179808

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Oostendorp RAB, Elvers JWH, van Trijffel E, Rutten GM, Scholten-Peeters GG, Heijmans M, et al. Relationships between context, process, and outcome indicators to assess quality of physiotherapy care in patients with whiplash-associated disorders: applying Donabedian's model of care. Pat Prefer Adher. (2020) 14:425–42. doi: 10.2147/PPA.S234800

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Oostendorp RAB, Elvers JWH, van Trijffel E, Rutten GM, Scholten-Peeters GG, Heijmans M, et al. Clinical characteristics and patient-reported outcomes of primary care physiotherapy in patients with whiplash-associated disorders: a longitudinal observational study. Pat Prefer Adher. (2020) 14:1733–50. doi: 10.2147/PPA.S262578

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Oostendorp RAB, Elvers JWH, van Trijffel E. The quality of physiotherapy care: development and application of quality indicators using scientific evidence and routinely collected data in the process of clinical reasoning. BFPT. (2019) 24:113–20. doi: 10.4103/bfpt.bfpt_4_19

CrossRef Full Text | Google Scholar

44. Bekkering GE, Hendriks HJM, Lanser K, Oostendorp RAB, Peeters GGM, Verhagen AP, et al. KNGF-richtlijn Whiplash. Ned Tijdschr Fysiother. (2001) 111:S1–25.

Google Scholar

45. Scholten-Peeters GG, Bekkering GE, Verhagen AP, Van Der Windt DA, Lanser K, Hendriks EJ, et al. Clinical practice guideline for the physiotherapy of patients with whiplash-associated disorders. Spine. (2002) 27:412–22. doi: 10.1097/00007632-200202150-00018

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Bier JD, Scholten-Peeters GGM, Staal JB, Pool J, Van Tulder M, Beekman E, et al. KNGF-Richtlijn Nekpijn. Amersfoort: Koninklijk Nederlands Genootschap voor Fysiotherapie (2016).

Google Scholar

47. Leigh TA. Best Practices Task Force. Clinical Practice Guidelines for the Physiotherapy Treatment of Whiplash-Associated Disorders. Vancouver, BC: Physiotherapy Association British Columbia (2004).

PubMed Abstract

48. Moore A, Jackson A, Jordan J, Hammersley S, Hill J, Mercer C, et al. Clinical Guidelines for the Physiotherapy Management of Whiplash-Associated Disorder (WAD). London: Chartered Society of Physiotherapy (2005).

Google Scholar

49. TRACsa. Clinical Guidelines for Best Practice Management of Acute and Chronic Whiplash Associated Disorders: Clinical Resource Guide. Adelaide, SA: South Australian Centre for Trauma and Injury Recovery (2008).

Google Scholar

50. Hurwitz EL, Carragee EJ, van der Velde G, Carroll LJ, Nordin M, Guzman J, et al. Bone and joint decade 2000–2010 task force on neck pain and its associated disorders. Treatment of neck pain: noninvasive interventions: results of the Bone and Joint Decade 2000–2010 Task Force on Neck Pain and Its Associated Disorders. Spine. (2008) 33:S123–52. doi: 10.1097/BRS.0b013e3181643f24

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Côté P, Wong JJ, Sutton D, Shearer HM, Mior S, Randhawa K, et al. Management of neck pain and associated disorders: a clinical practice guideline from the Ontario Protocol for Traffic Injury Management (OPTIMa) Collaboration. Eur Spine J. (2016) 25:2000–22. doi: 10.1007/s00586-016-4467-7

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Bussières AE, Stewart G, Al-Zoubi F, Decina P, Descarreaux M, Hayden J, et al. The treatment of neck pain-associated disorders and Whiplash-associated disorders: a clinical practice guideline. J Manipul Physiol Ther. (2016) 39:523–64. doi: 10.1016/j.jmpt.2016.08.007

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Jones MA, Jensen G, Edwards I. Clinical reasoning in physiotherapy. In: Higgs J, Jones M, Loftus S, Christensen N, editors. Clinical Reasoning in the Health Professions. Edinburgh: Butterworth Heinemann (2008). p. 193–204.

54. Oostendorp RAB, Elvers JWH, Van Trijffel E. We are missing more. An international measurable model of clinical reasoning using quality indicators and routinely collected data. J Man Manip Ther. (2019) 27:253–7. doi: 10.1080/10669817.2019.1648714

PubMed Abstract | CrossRef Full Text

55. Donabedian A. Evaluating the quality of medical care 1966. Milbank Q. (2005) 83:691–729. doi: 10.1111/j.1468-0009.2005.00397.x

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Donabedian A. Methods for deriving criteria for assessing the quality of medical care. Med Care Rev. (1980) 37:653–98.

PubMed Abstract | Google Scholar

57. Donabedian A. The quality of care. How can it be assessed? JAMA. (1988) 260:1743–8. doi: 10.1001/jama.1988.03410120089033

PubMed Abstract | CrossRef Full Text | Google Scholar

58. World Health Organization. International Classification of Functioning, Disability and Health. Geneva: WHO (2001).

Google Scholar

59. Neeleman-van der Steen CWM, Ven G van de, Krol MW, Bie RA de, Oostendorp RAB, Braspenning JC. Prestatie-indicatoren Fysiotherapie. Het Ontwikkelen en Testen van een Basisset van Publieke Kwaliteitsindicatoren voor de Fysiotherapie. Nijmegen; Maastricht: Radboud Universiteit Nijmegen en Universiteit Maastricht (2009).

Google Scholar

60. Scholte M. Developing Quality Indicators for Physical Therapy. PhD Thesis It Takes Three to Tango: Developing and Implementing Quality Indicators for Physical Therapy: Lessons Learned. Nijmegen: Radboud University Nijmegen (2017). p. 33–48.

Google Scholar

61. Milholland AV, Wheeler SG, Heieck JJ. Medical assessment by a Delphi group opinion technic. N Engl J Med. (1973) 288:1272–5. doi: 10.1056/NEJM197306142882405

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Scholten RJPM, Tuut MK, Kremer LCM, Assendelft WJJ. Beoordelen van de kwaliteit van medisch-wetenschappelijk onderzoek. In: Van Everdingen JJE, Burgers JS, Assendelft WJJ, Swinkels JA, Van Barneveld TA, Van de Klundert JLM, editors. Evidence-Based Richtlijnontwikkeling. Houten: Bohn Stafleu Van Loghum (2004). p. 158–71.

Google Scholar

63. Higgs J, Jones M. Clinical decision making and multiple problem spaces. In: Higgs J, Jones M, Loftus S, Christensen N, editors. Clinical reasoning in the health profressions. Edinburgh: Butterworth Heinemann (2008). p. 3–18.

Google Scholar

64. Jones MA, Rivett DA. Introduction to clinical reasoning. In: Jones MA, Rivett DA, editors. Clinical Reasoning for Manual Therapists. Edinburgh: Butterworth Heinemann (2004). p. 3–24.

Google Scholar

65. Squires B, Gargan MF, Bannister GC. Soft-tissue injuries of the cervical spine. 15-year follow-up. J Bone Joint Surg Br. (1996) 78:955–7. doi: 10.1302/0301-620X.78B6.0780955

CrossRef Full Text | Google Scholar

66. Kasch H, Qerama E, Kongsted A, Bendix T, Jensen TS, Bach FW. Clinical assessment of prognostic factors for long-term pain and handicap after whiplash injury: a 1-year prospective study. Eur J Neurol. (2008) 15:1222–30. doi: 10.1111/j.1468-1331.2008.02301.x

PubMed Abstract | CrossRef Full Text | Google Scholar

67. Åsenlöf P, Bring A, Söderlund A. The clinical course over the first year of whiplash associated disorders (WAD): pain-related disability predicts outcome in a mildly affected sample. BMC Musculoskelet Disord. (2013) 14:361. doi: 10.1186/1471-2474-14-361

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Styrke J, Sojka P, Björnstig U, Stålnacke BM. Symptoms, disabilities, and life satisfaction five years after whiplash injuries. Scand J Pain. (2014) 5:229–36. doi: 10.1016/j.sjpain.2014.06.001

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Myrtveit SM, Carstensen T, Kasch H, Ørnbøl E, Frostholm L. Initial healthcare and coping preferences are associated with outcome 1 year after whiplash trauma: a multicentre 1-year follow-up study. BMJ Open. (2015) 5:e007239. doi: 10.1136/bmjopen-2014-007239

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Gehrt TB, Wisbech Carstensen TB, Ørnbøl E, Fink PK, Kasch H, Frostholm L. The role of illness perceptions in predicting outcome after acute whiplash trauma: a multicenter 12-month follow-up study. Clin J Pain. (2015) 31:14–20. doi: 10.1097/AJP.0000000000000085

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Van Trijffel E, Oostendorp RAB, Elvers JWH. Routinely collected data as real-world evidence for physiotherapy practice. Physiother Theory Pract. (2019) 35:805–9. doi: 10.1080/09593985.2019.1615678

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Hoque DME, Ruseckaite R, Lorgelly P, McNeil JJ, Sue M Evans SM. Cross-sectional study of characteristics of clinical registries in Australia: a resource for clinicians and policy makers. Int J Qual Health Care. (2018) 30:192–9. doi: 10.1093/intqhc/mzx196

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Maissan F, Pool J, de Raaij E, Mollema J, Ostelo R, Wittink H. The clinical reasoning process in randomized clinical trials with patients with non-specific neck pain is incomplete: a systematic review. Musculoskelet Sci Pract. (2018) 35:8–17. doi: 10.1016/j.msksp.2018.01.011

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Saturni S, Bellini F, Braido F, Paggiaro P, Sanduzzi A, Scichilone N, et al. Randomized controlled trials and real-life studies. Approaches and methodologies: a clinical point of view. Pulm Pharmacol Ther. (2014) 27:129–38. doi: 10.1016/j.pupt.2014.01.005

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. RECORD Working Committee. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. (2015) 12:e1001885. doi: 10.1371/journal.pmed.1001885

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. (2014) 4: MR000034. doi: 10.1002/14651858.MR000034.pub2

PubMed Abstract | CrossRef Full Text | Google Scholar

77. Franklin JM, Schneeweiss S. When and how can real world data analyses substitute for randomized controlled trials? Clin Pharm Ther. (2017) 102:924–33. doi: 10.1002/cpt.857

PubMed Abstract | CrossRef Full Text | Google Scholar