Usability of an Educational Intervention to Overcome Therapeutic Inertia in Multiple Sclerosis Care

Background: Educational interventions are needed to overcome knowledge-to-action gaps in clinical care. We previously tested the feasibility and potential efficacy of an educational intervention that facilitates treatment decisions in multiple sclerosis care. A demonstration of the usability of such an intervention is crucial prior to demonstration of efficacy in a large trial.

Objectives: To evaluate the usability of a novel, pilot-tested intervention aimed at neurologists to improve therapeutic decisions in multiple sclerosis (MS) care.

Methods: We surveyed 50 neurologists from Chile, Argentina, and Canada randomized to an educational intervention arm of a pilot feasibility study using the System Usability Score (SUS) to assess the usability of a traffic light system (TLS)-based educational intervention. The TLS facilitates therapeutic decisions, allowing participants to easily recognize high-risk scenarios requiring treatment escalation. The SUS is a validated 10-item questionnaire with five response options. The primary outcome was the average and 95% confidence interval (CI) of the SUS score. Values above 68 are considered highly usable.

Results: Of 50 neurologists invited to be part of the study, all completed the SUS scale and the study. For the primary outcome, the average usability score was 74.7 (95%CI 70.1–79.2). There was one outlier with a score of 35. The usability score excluding the outlier was 76.8 (95%CI 72.7–80.8). Multivariate analysis revealed no association between participants' characteristics and the SUS score.

Conclusions: Our educational intervention has shown high usability among neurologists. The next step is to evaluate the effectiveness of this educational intervention in facilitating treatment decisions for the management of multiple sclerosis in a large trial.

Background

Recent studies have shown many patients with multiple sclerosis (MS) remain undertreated (1, 2). Therapeutic inertia (TI) is defined as the lack of treatment initiation or escalation when treatment goals are unmet (3, 4). Approximately 70–80% of neurologists caring for patients with MS are estimated to be affected by TI (2). Physician factors are most commonly related to TI (e.g., low tolerance to uncertainty, status quo bias) (5, 6).

Educational strategies are developed in medicine to either fill knowledge gaps (where there is limited background information on the management of a medical condition) or knowledge-to-action gaps (i.e., failure to integrate background knowledge into a diagnostic or therapeutic decision). To the best of our knowledge, there are no such interventions aimed at influencing clinician decisions to ameliorate the consequences of TI (e.g., poor patient outcomes) (2–4). In a previous study, we showed the feasibility and potential efficacy of an educational intervention using a traffic light system (TLS) to overcome TI in the management of MS (7). The TLS emerged as a warning and risk categorization strategy to reduce human errors (8). It facilitates the decision-making process relying on a “hard wired” process that applies the traffic light colors to match three types of situations: red light (“high risk”/“stop and think”), yellow light (“warning”/“reassess in a period of time”) and green light (“stable”/“continue the same strategy”).

The TLS aids the integration of specific situations with an action (9, 10). For example, studies have shown that the TLS can help physicians select the course of action for children presenting with fever or facilitate healthier choices by signaling with a red color food with high content of sugar, cholesterol, and sodium (11). However, many educational interventions or scores are not widely used or implemented in clinical practice. Consequently, it is crucial to determine the potential usability of an educational intervention prior to testing its efficacy in large and expensive randomized clinical trials.

In the present study, we evaluated the usability of our educational intervention among neurologists caring for MS patients. We hypothesized that our educational intervention using the TLS has high usability, as defined by the system usability score (SUS), in this population.

Methods

We included neurologists participating in an international (Chile, Argentina, Canada), parallel-group, randomized pilot study evaluating the feasibility of the educational intervention (TLS: active group) (Figure 1) compared to usual care (control group) in the management of MS (7). The recruitment of participants was facilitated by the National Neurological Societies (MS Society for Canada) of the three participating countries providing mailing lists of non-specialized neurologists and those with MS expertise. Participants randomized to the educational intervention were exposed to 10 case-scenarios and asked to make a therapeutic choice, guided by the TLS-based education intervention. Following these case-scenarios, they were asked to assess the utility of the TLS using the SUS. Given our previous findings (e.g., association between physicians' risk preferences and aversion to ambiguity with TI), we also included previously tested experiments to assess participants' risk preferences and aversion to ambiguity to evaluate their association with the SUS (2). In brief, participants were asked about the minimal certain payoff they would prefer over the equiprobable gamble of winning 400 or 0 US$ (expected value of 200 US$). Ambiguity aversion is defined as dislike for events with unknown probability over events with known probability. Participants had to choose between two options: (a) a 50% chance of winning 400 vs. 0 US$ or (b) an unknown probability of winning 400 US$ (where the unknown probability of winning may be higher or lower than 50%). Further details of the protocol were published in ClinicalTrials.gov # NCT03134794 and elsewhere (2, 7).

FIGURE 1

Figure 1. Educational intervention: the traffic light system may facilitate therapeutic decisions in MS care (7). Participants viewed the two informative panels (A,B) and a third panel providing an example (C). (A) Introduction of the traffic light system. (B) Implementation of the traffic light system. (C) Example application of the traffic light system.

The System Usability Score (SUS)

The SUS is a validated 10-item questionnaire with a 5-point Likert rating scale that ranges from 1 = strongly disagree to 5 = strongly agree. Questions with positive and negative connotation are alternated to avoid any biases. A score is calculated by subtracting participants' responses from 5 for even questions and by subtracting one from responders' answers for odd questions. This procedure scales all values from 0 to 4 (with four being the most positive response). The final score is obtained by adding the converted responses for each participant and multiplying the total by 2.5. As a result, the possible values range from 0 to 100 (12). A score of 68 or higher is considered “highly acceptable” (13–16). The SUS has been validated in different populations and countries and used in over 500 instruments (13–16).

Educational Intervention (Figure 1): The TLS

Our educational intervention is based on the application of the TLS to medical-decision making (9, 10, 17). The TLS was intended to assist participants in identifying high-risk case-scenarios, where MS patients had both clinical relapses and evidence of radiological activity. Our educational intervention focused on MS because of tiered therapeutic options and consensus about treatment escalation (e.g., presence of clinical relapses and radiological activity) (18, 19). Our educational intervention was delivered in a single presentation. The total duration of the educational intervention was <5 min (Figure 1) (7).

Participants were asked to select the traffic light that best matched the case-scenario based on clinical relapse history and results from brain imaging. For example, a “red” traffic light may have been selected for a patient who had both a clinical relapse and some level of activity in brain imaging (Figure 2), where treatment escalation is likely the correct decision. Similarly, participants may have selected the color “yellow” (representing caution requiring a reassessment within 6–12 months) when case-scenarios had either a clinical relapse or some degree of activity in brain imaging (but not both). At the end of the study, participants were asked to evaluate the delivered educational intervention using the validated SUS instrument. Further details were published in the pilot study (7).

FIGURE 2

Figure 2. Usability Score System. The SUS is a 10 point-item questionnaire with 5 response options.

Treatment Options

The treatment options for relapsing-remitting MS (RRMS) include first-line (beta interferons, glatiramer acetate, etc.), second-line (fingolimod), and third-line (natalizumab, alemtuzumab) therapies (20). Other recently approved agents were not included if they were not available in the participating countries at the time of the study (e.g., Ocrelizumab, Cladribine, etc.). For the present analysis, we used the aforementioned scheme according to the current clinical practice (2, 21, 22).

Outcome Measures

The primary outcome was the average SUS. Values >68 points are considered to reflect high usability (13, 14). The secondary outcome was the proportion of participants with a usability score >68 points.

Statistical Analysis

Given the pilot nature of this study, we performed primarily descriptive statistics for the primary and secondary outcomes. As a secondary analysis, we developed a linear regression model to explore factors associated with the usability score including age, gender, general neurologist vs. MS specialist status, years of experience, practice setting and volume, physicians' risk preferences and aversion to ambiguity, and authorship in a recent peer-review publication. Variables entered into the models were defined a priori based on the results of our previous studies (e.g., MS specialist vs. general neurologists, physicians' risk preferences and aversion to ambiguity) adjusted for age and sex (7). Specialist status was self-defined. Collinearity among variables was explored by variance inflation factors (VIFs). Values below 10 are considered a good indicator of lack of collinearity. All tests were 2-tailed, and p < 0.05 was considered significant. We used STATA 13 (College Station, TX: StataCorp LP) to conduct all analyses.

The study was approved by the Research Ethics Board of St. Michael's Hospital, University of Toronto, Canada.

Results

Of 50 neurologists who were invited to participate in the study, all (100%) completed the usability scale. Overall, the mean (SD) age was 45.6 (± 11.1) years; 21 (42%) were female neurologists. One third of participants primarily focused their practice on MS care. Table 1 summarizes baseline characteristics of the study population.

TABLE 1

Table 1. Baseline characteristics of participants.

For the primary outcome, the average usability score [95% confidence intervals (CI)] was 74.7 (95%CI 70.1–79.2). There was one outlier with a score of 35. The usability score excluding that outlier was 76.8 (72.7–80.8).

For the secondary outcome, 70% (n = 35) of participants achieved a usability score >68 points. SUS percentiles are shown in Figure 3.

FIGURE 3

Figure 3. Percentiles of the Usability Score System (SUS).

Multivariate regression analysis revealed no statistically significant association between the variables tested and SUS score (Table 2). There was no collinearity among the included variables as reflected by VIF scores < 1.5. Overall, the included variables only explained <9% of the variability of the SUS scores (r² = 0.088).

TABLE 2

Table 2. Multivariate analysis: factors associated with the SUS score.

Discussion

In the present study, we found a high level of usability of a TLS-based educational intervention aimed at overcoming TI in the management of MS. Our results suggest that a TLS-based intervention is amenable to being widely used in clinical practice. We identified no specific factors associated with higher usability scores, suggesting that its use may not be affected by age, gender, expertise, years of practice, volume of MS patients seen per week or practice setting. The usability of the TLS was not influenced by individual physicians' risk preferences or aversion to ambiguity.

The use of the TLS is a novel way to optimize treatment decisions. It has been successfully applied to different medical fields, including the selection of healthier food choices leading to weight loss or the detection of children with fever at high risk of developing a serious bacterial infection (9, 10). In a previous randomized trial, we pilot-tested the application of the TLS to determine the feasibility (and preliminary efficacy data for sample size calculation for a larger trial) in treatment decisions in MS care. We showed feasibility and promising results in reducing TI (OR 0.57; 95% CI 0.26–1.22) (7).

Our findings have practical clinical implications. Our results showed excellent usability scores for a simple and brief educational intervention that helps categorize MS patients according to their future risk of disease progression. This educational strategy was designed to ameliorate TI by overcoming knowledge-to-action gaps. Physicians who are resistant to treatment changes (e.g., status quo) when indicated by best practice guidelines may benefit from using the TLS.

Our results nicely integrate with our previous work showing one of the major components influencing TI is aversion to risk and ambiguity (defined as the probability of an event being unknown) (2, 7). To the best of our knowledge, there are no other proven educational interventions to address this phenomenon. Aversion to risk and ambiguity are concepts derived from studies in behavioral economics to characterize individuals and describe their decisions. We previously showed that aversion to ambiguity is an independent predictor of physicians' TI (2). The use of the TLS may help such participants to make prompt decisions and overcome TI.

Our study has some limitations. First, the sample size is small given the pilot design. Second, we used a single measurement of usability. However, the SUS has been validated and extensively applied to over 500 interventions (13–15). Finally, we included participants from three specific countries. It is possible that these findings may not be generalizable to clinicians from other countries.

Despite these limitations, our study suggests that a simple and brief educational intervention applying the TLS may have high potential for use by neurologists when evaluating therapeutic choices for MS patients. Our findings should be put into perspective as the TLS may facilitate early identification of high-risk patients that require a therapeutic change. For example, clinicians may apply the TLS for patients waiting to be seen in the outpatient clinics by adding a color tag to those screened as high-risk according to the medical situation (“red color tag” for MS patients with new symptoms suggestive of a clinical relapse and radiological progression). This approach may help planning resources, facilitate timely discussions and prioritized clinical outcomes when treatment escalation is indicated (23). Although our intervention was designed to facilitate therapeutic decisions in MS care, it can be applied to decisions in other neurological fields (e.g., epilepsy, migraine) and chronic conditions (e.g., hypertension, diabetes). Given that we have tested the feasibility and usability of this promising educational intervention, the next step is the assessment of its efficacy in affecting management decisions and resultant clinical outcomes in a large, properly designed randomized clinical trial.

Ethics Statement

This study was carried out in accordance with the recommendations of the Research Ethics Board of St. Michael's Hospital, University of Toronto, Canada. The protocol was approved by the Research Ethics Board at St Michael's Hospital. All subjects gave informed consent in accordance with the Declaration of Helsinki.

Author Contributions

GS: study concept and design, creation of the educational intervention, acquisition of data, analysis and interpretation of the data, and obtaining funding. PT and CR: study concept and design, interpretation of the data, critical revision of the manuscript for intellectual content, and study supervision. FC: interpretation of the data, and critical revision of the manuscript for intellectual content. MT: study implementation, interpretation of the data, and critical revision of the manuscript for intellectual content. JO and MF: interpretation of the data, critical revision of the manuscript for intellectual content. JM: study concept and design, interpretation of the data, critical revision of the manuscript for intellectual content. XM: supervision of MS case-scenarios, interpretation of the data, critical revision of the manuscript for intellectual content. MM: study concept and design, design of the educational intervention, interpretation of the data, critical revision of the manuscript for intellectual content.

Funding

The study was sponsored by the Sociedad Española de Neurologia (SEN), Sociedad Argentina de Neurologia, Instituto de Neurociencias Buenos Aires (INEBA) and partially funded by an operating grant from Roche Farma. The sponsors were not involved in the design, execution, analysis, interpretation, or reporting of the results.

Conflict of Interest Statement

PT and CR were funded by the Swiss National Science Foundation (PNT: PP00P1_150739, CRSII3_141965, and 100014_165884, CCR: 105314_152891, CRSII3_141965, and 320030_143443). GS is supported by the Heart and Stroke Foundation Career Award following a peer-reviewed open competition. JM was employed by company Roche Pharma.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors are most grateful to all physicians participating in the study. We appreciate the support from the Department of Economics at the University of Zurich by facilitating access to the Qualtrics platform. We also appreciate the support of Dr. Ryan Brydges and Nikki Woods of the Wilson Center for Medical Education at the University of Toronto.

References

1. Ziemssen T, Derfuss T, de Stefano N, Giovannoni G, Palavra F, Tomic D, et al. Optimizing treatment success in multiple sclerosis. J Neurol. (2016) 263:1053–65. doi: 10.1007/s00415-015-7986-y

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Saposnik G, Sempere AP, Prefasi D, Selchen D, Ruff CC, Maurino J, et al. Decision-making in multiple sclerosis: the role of aversion to ambiguity for therapeutic inertia among neurologists (DIScUTIR MS). Front Neurol. (2017) 8:65. doi: 10.3389/fneur.2017.00065

PubMed Abstract | CrossRef Full Text | Google Scholar

3. O'Connor PJ, Sperl-Hillen JAM, Johnson PE, Rush WA, Biltz G. Clinical inertia and outpatient medical errors. In: Henriksen K, Battles JB, Marks ES, Lewin DI, editors. Advances in Patient Safety: From Research to Implementation, Vol. 2: Concepts and Methodology. Rockville, MD (2005). p. 293–308.

Google Scholar

4. Phillips LS, Branch WT, Cook CB, Doyle JP, El-Kebbi IM, Gallina DL, et al. Clinical inertia. Ann Intern Med. (2001) 135:825–34. doi: 10.7326/0003-4819-135-9-200111060-00012

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Reach G. Patient non-adherence and healthcare-provider inertia are clinical myopia. Diabetes Metab. (2008) 34:382–5. doi: 10.1016/j.diabet.2008.02.008

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Saposnik G, Montalban X. Therapeutic Inertia in the new landscape of multiple sclerosis care. Front Neurol. (2018) 9:174. doi: 10.3389/fneur.2018.00174

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Saposnik G, Maurino J, Sempere AP, Terzaghi MA, Ruff CC, Mamdani M, et al. Overcoming therapeutic inertia in multiple sclerosis care: a pilot randomized trial applying the traffic light system in medical education. Front Neurol. (2017) 8:430. doi: 10.3389/fneur.2017.00430

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Orriols L, Delorme B, Gadegbeku B, Tricotel A, Contrand B, Laumonet B, et al. Prescription medicines and the risk of road traffic crashes: a French registry-based study. PLoS Med. (2010) 7:e1000366. doi: 10.1371/journal.pmed.1000366

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Sonnenberg L, Gelsomin E, Levy DE, Riis J, Barraclough S, Thorndike AN. A traffic light food labeling intervention increases consumer awareness of health and healthy choices at the point-of-purchase. Preventive Med. (2013) 57:253–7. doi: 10.1016/j.ypmed.2013.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Murphy MS, Baker M. When used appropriately, NICE's traffic light system for febrile children helps clinicians make safe and appropriate decisions. BMJ (2014) 348:g2518. doi: 10.1136/bmj.g2518

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Enax L, Krajbich I, Weber B. Salient nutrition labels increase the integration of health attributes in food decision-making. Judgment Decision Making (2016) 11:460–71. Available online at: http://psycnet.apa.org/record/2016-49579-005

Google Scholar

12. Lewis JR, Sauro J. The factor structure of the system usability scale. In: Kurosu M, editor Human Centered Design, Berlin, Heidelberg: Springer. (2009) 94–103. doi: 10.1007/978-3-642-02806-9_12

CrossRef Full Text | Google Scholar

13. Brooke J. SUS: a retrospective. J Usability Stud. (2013) 8:29–40.

Google Scholar

14. Brooke J. SUS-A Quick and Dirty Usability Scale. Usability Evaluation in Industry. Rockville, MD: CRC Press (1996).

15. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. (2009) 4:114–23.

Google Scholar

16. Gardner CL, Liu F, Fontelo P, Flanagan MC, Hoang A, Burke HB. Assessing the usability by clinicians of VISION: a hierarchical display of patient-collected physiological information to clinicians. BMC Med Inform Decis Mak. (2017) 17:41. doi: 10.1186/s12911-017-0435-3

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Hawley KL, Roberto CA, Bragg MA, Liu PJ, Schwartz MB, Brownell KD. The science on front-of-package food labels. Public Health Nutr. (2013) 16:430–9. doi: 10.1017/S1368980012000754

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Sormani MP, Gasperini C, Romeo M, Rio J, Calabrese M, Cocco E, et al. Assessing response to interferon-beta in a multicenter dataset of patients with MS. Neurology (2016) 87:134–40. doi: 10.1212/WNL.0000000000002830

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Filippi M, Rocca MA, Ciccarelli O, De Stefano N, Evangelou N, Kappos L, et al. MRI criteria for the diagnosis of multiple sclerosis: MAGNIMS consensus guidelines. Lancet Neurol. (2016) 15:292–303. doi: 10.1016/S1474-4422(15)00393-2

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Tramacere I, Del Giovane C, Salanti G, D'Amico R, Filippini G. Immunomodulators and immunosuppressants for relapsing-remitting multiple sclerosis: a network meta-analysis. Cochrane Database Syst Rev. (2015) 2015:CD011381. doi: 10.1002/14651858.CD011381.pub2

CrossRef Full Text | Google Scholar

21. Prosperini L, Gianni C, Leonardi L, De Giglio L, Borriello G, Galgani S, et al. Escalation to natalizumab or switching among immunomodulators in relapsing multiple sclerosis. Mult Scler. (2012) 18:64–71. doi: 10.1177/1352458511417481

CrossRef Full Text | Google Scholar

22. Montalban X, Gold R, Thompson AJ, Otero-Romero S, Amato MP, Chandraratna D, et al. ECTRIMS/EAN guideline on the pharmacological treatment of people with multiple sclerosis. Mult Scler. (2018) 24:96–120. doi: 10.1177/1352458517751049

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Day GS, Rae-Grant A, Armstrong MJ, Pringsheim T, Cofield SS, Marrie RA. Identifying priority outcomes that influence selection of disease-modifying therapies in MS. Neurology (2018). doi: 10.1212/CPJ.0000000000000449

CrossRef Full Text | Google Scholar