Harnessing Real-World Data to Inform Decision-Making: Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS PATHS)

Background: Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS PATHS) is the first example of a learning health system in multiple sclerosis (MS). This paper describes the initial implementation of MS PATHS and initial patient characteristics.

Methods: MS PATHS is an ongoing initiative conducted in 10 healthcare institutions in three countries, each contributing standardized information acquired during routine care. Institutional participation required the following: active MS patient census of ≥500, at least one Siemens 3T magnetic resonance imaging scanner, and willingness to standardize patient assessments, share standardized data for research, and offer universal enrolment to capture a representative sample. The eligible participants have diagnosis of MS, including clinically isolated syndrome, and consent for sharing pseudonymized data for research. MS PATHS incorporates a self-administered patient assessment tool, the Multiple Sclerosis Performance Test, to collect a structured history, patient-reported outcomes, and quantitative testing of cognition, vision, dexterity, and walking speed. Brain magnetic resonance imaging is acquired using standardized acquisition sequences on Siemens 3T scanners. Quantitative measures of brain volume and lesion load are obtained. Using a separate consent, the patients contribute DNA, RNA, and serum for future research. The clinicians retain complete autonomy in using MS PATHS data in patient care. A shared governance model ensures transparent data and sample access for research.

Results: As of August 5, 2019, MS PATHS enrolment included participants (n = 16,568) with broad ranges of disease subtypes, duration, and severity. Overall, 14,643 (88.4%) participants contributed data at one or more time points. The average patient contributed 15.6 person-months of follow-up (95% CI: 15.5–15.8); overall, 166,158 person-months of follow-up have been accumulated. Those with relapsing–remitting MS demonstrated more demographic heterogeneity than the participants in six randomized phase 3 MS treatment trials. Across sites, a significant variation was observed in the follow-up frequency and the patterns of disease-modifying therapy use.

Conclusions: Through digital health technology, it is feasible to collect standardized, quantitative, and interpretable data from each patient in busy MS practices, facilitating the merger of research and patient care. This approach holds promise for data-driven clinical decisions and accelerated systematic learning.

Introduction

The multiple sclerosis (MS) treatment landscape has experienced a dramatic evolution over the past two decades; there are now 16 approved disease-modifying therapies (DMTs). Approaches to personalized medicine in MS, however, have not kept pace. Defining disease prognosis, treatment outcomes, monitoring treatment response, and determining optimal treatment sequencing remain variable and somewhat subjective in MS practice. Personalized medicine efforts have focused both on identifying informative patient phenotypes (1) and effectively integrating and visualizing individual patient data (2).

The rigorous collection and analysis of real-world data may accelerate the development of personalized medicine in MS and address some of these gaps (3–5). Opportunities and challenges related to data pooling and data standardization in MS were recognized by early pioneers, beginning decades ago (6, 7), and recent progress has been summarized (8, 9). These efforts have ushered in an era of data standardization and pooling in an attempt to extend systematic learning beyond structured research protocols to more representative real-world populations. For maximum impact, data should include standardized and quantitative clinical, radiologic, and biological phenotyping from a heterogeneous population representative of the diversity of patients with MS seen in everyday clinical practice.

The learning health system (LHS) model, as proposed by the Institute of Medicine, outlines a method to enable broad-scale quantitative patient phenotyping through the merging of clinical research and healthcare delivery (4, 10). The LHS (also known as evidence-generating medicine) seeks to produce better outcomes and research based on real-time data acquisition (10). The tenets of the LHS concept include collection of standardized, meaningful data on every patient seeking care, increased engagement of patients in the process of care, use of quantitative data for clinical decision-making, and aggregation of data from populations for systematic learning (10). The LHS represents a culture of continuous learning, feeding insights back into care delivery to continuously reanalyze, revalidate, and improve outcomes.

The current report describes the first LHS in MS, Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS PATHS). In MS PATHS, quantitative clinical and imaging data are collected in a standardized manner for each patient as part of routine care. Data are collected as part of the patient's clinical evaluation, while—with a patient's permission—the data are pseudonymized and aggregated for systematic learning. The program leverages technology and patient engagement to automate data collection and analysis, minimizing the burden on the care system and providers. The specific goals of MS PATHS are to better understand the disease, identify the predictors of therapeutic responses, define and measure outcomes during the course of care, and develop approaches to personalized medicine.

We aim to demonstrate the feasibility of implementing the key features of an LHS in MS by describing the patient population enrolled in MS PATHS to date, initial data completion rates, differences between MS PATHS and typical MS clinical trial patients, heterogeneity of practice across the network, and utilization of the research data.

Materials and Methods

Initial Design

MS PATHS is a collaborative network of healthcare institutions that have standardized elements of their clinical assessments and collaborated with Biogen to implement a centralized database for research purposes. This network was designed based on the guiding principles described in Table 1. In 2014, a group of stakeholders from Cleveland Clinic, Johns Hopkins University, New York University, and Biogen began the planning process to best design the tools, systems, and governance needed for an LHS in MS.

TABLE 1

Table 1. Guiding principles for MS PATHS.

Methods for standardized clinical and imaging data collection that do not increase the burden on providers or generate significant incremental cost were an important aspect of the planning process. Technology was developed to allow patient self-administered clinical assessment, resulting in standardized, high-quality clinical data. Technician-based testing—specifically, Multiple Sclerosis Functional Composite testing—was adapted to a series of patient self-administered iPad-based tests to provide quantitative data related to processing speed, low-contrast visual acuity, manual dexterity, and walking speed (11). This was done in order to facilitate the neuroperformance testing for every patient with MS, even in a busy clinical practice (12–14). The iPad-based clinical assessment tool, called the Multiple Sclerosis Performance Test (MSPT), also enabled the collection of a structured patient history.

Because the clinical assessment was patient self-administered, there are differences between the MSPT and the physician-derived measures used in clinical trials and traditional MS practice registries. Neurologist-determined relapses and Expanded Disability Status Scale were replaced by patient-reported relapse and Patient Determined Disease Steps. Prior studies support the validity of self-reported relapses (15, 16) and showed a strong correlation between Patient Determined Disease Steps and Expanded Disability Status Scale (17–19). The MSPT also enabled the collection of quality-of-life data. The Quality of Life in Neurological Disorders (Neuro-QoL) measure was selected as a standardized, well-validated patient-reported outcome instrument (20–22). The computer-adapted version of Neuro-QoL was incorporated into the MSPT to minimize administration time.

Two standardized magnetic resonance imaging (MRI) acquisition sequences were incorporated into routine MS imaging protocols to enable the reliable calculation of brain atrophy and lesions at the point of care. Through a collaboration with Siemens Healthineers, a software prototype is currently in development, with a focus on workflow integration and performance adequate for individual patient clinical decision-making.

To enable translational research, an MS PATHS research substudy was implemented to enable the collection of blood samples under a research protocol that could be linked to the standardized clinical and imaging data for future analyses.

Participating Healthcare Institutions

To be eligible for MS PATHS, the participating healthcare institutions had to have an MS center with an active census of ≥500 patients that routinely used functional measures and MRI in their clinical practice and a willingness to further standardize aspects of their clinical and radiological assessments. In addition, each participating healthcare institution had to have at least one Siemens 3T MRI scanner available for clinical use. Each participating center had to be willing to implement a centralized health information exchange architecture to transfer the data to a research database and obtain approval of the project protocol by the institutional review board or ethics committee, information technology (IT) security, and/or data privacy committee and agree to adhere to Good Clinical Practice and ethical principles as outlined in the Declaration of Helsinki. The participating healthcare institutions of each investigator receive financial compensation for data shared and biosamples collected as part of this program based on fair market value.

Patients

Patients with a confirmed diagnosis of MS, including clinically isolated syndrome, and the ability to understand the purpose and the risks of the project are eligible to enroll in MS PATHS. In contrast with a traditional prospective observational study, the sites in MS PATHS agree to adopt the outlined standard of care, and the patients provide authorization for the use of protected health information in accordance with national and local subject privacy regulations. Authorization format was determined by the local institutional review board or ethics committee and ranges from the use of a standard medical information privacy waiver (four institutions) to an oral consent (one institution) to a full informed consent (five institutions). The investigators and research coordinators are encouraged to invite all patients at each MS center to participate. At steady state, the network aims to have 80% of the MS patients at each participating MS center enrolled in MS PATHS.

Procedures

Upon enrollment, the patients are assigned a unique MS PATHS identification number that acts as the patient identifier in the LHS. This allows linkage of pseudonymized data from different sources for the same patient. The authorization form allows for prospective data sharing as well as sharing of data from 12 months before the date of consent.

The data elements collected routinely for patients in MS PATHS are listed in Table 2. Clinical data are collected using the MSPT. In addition, structured clinical data are shared from electronic medical records at each institution. Imaging data include brain MRIs acquired using two standardized sequences (three-dimensional fluid-attenuated inversion recovery and three-dimensional T1 acquired on Siemens 3T scanners; Table 3), information from radiology reports, and quantitative measures of brain atrophy and lesion metrics derived from an image analysis software prototype.

TABLE 2

Table 2. Current data elements collected in the MS PATHS learning health system.

TABLE 3

Table 3. Standardized Siemens 3T brain MRI sequence parameters for MS PATHS.

Data collected during patient care visits can also be linked to substudies. Each substudy requires a separate protocol and informed consent to allow for research assessments or procedures and for the linkage of the substudy data with patient care data. An example is an ongoing biorepository substudy that collects a one-time blood sample for future genomic analyses (10 ml for adult patients and 6 ml for pediatric patients) and repeated blood samples for future biomarker analyses (33 ml for adult patients and 13.5 ml for pediatric patients) not more frequently than every 6 months. The clinical phenotyping information for any genetic or biomarker analyses will be derived from the routine clinical care data that are shared as part of MS PATHS.

Governance

The governance for MS PATHS requires a structure and a process that:

(1) Are fair, transparent, and compliant for all participating organizations;

(2) Assure confidence from all participating organizations;

(3) Foster collaboration;

(4) Reinforce the clinical and research integrity of participating healthcare institutions;

(5) Ensure that all participating organizations have sufficient freedom to operate; and

(6) Foster innovation, limit risk, and enable long-term success.

A steering committee was tasked with:

(1) Providing strategic and operational guidance;

(2) Setting the MS PATHS scientific strategy;

(3) Creating and overseeing data and sample access rules;

(4) Monitoring performance; and

(5) Providing other oversight activities as needed.

The steering committee consists of six representatives from participating healthcare institutions and one representative from Biogen. Three steering committee seats are reserved for investigators from the institutions that collaborated with Biogen on the initial planning process (Cleveland Clinic Foundation, Johns Hopkins University, and New York University), and one seat each is designated for an additional US healthcare institution, an EU healthcare institution, and a radiologist in MS PATHS.

The steering committee governs a set of data and sample access subcommittees responsible for approving requests for MS PATHS data and samples. The healthcare institutions and Biogen each have separate data and sample access committees. Biogen employees do not sit on the healthcare institution committees and vice versa. Separate subcommittees for Biogen and participating healthcare institutions were set up to ensure the independence of research led by the participating healthcare institutions. The healthcare institution and Biogen subcommittees operate under the same procedures, and all approved uses of the data or the samples are posted on the MS PATHS research website, which is accessible to Biogen and the participating healthcare institutions to ensure transparency and promote collaboration. In addition to approving data or sample requests, these subcommittees are also responsible for reviewing resulting presentations or publications to ensure alignment with the original request. Biogen, as the sponsor, has no role in writing or editing publications unless a Biogen employee is a co-author.

Standardization

Participating healthcare institutions have implemented the MSPT (11, 13, 14), an iPad-based medical assessment tool that quantifies major MS-associated motor, visual, and cognitive symptoms and quality-of-life outcomes. The MSPT incorporates a structured patient history (gathers the patient's relevant demographic and socioeconomic information, MS history, MS treatment information, and self-reported disability using the Patient Determined Disease Steps) (17–19), 12 subscales of the Neuro-QoL (20–22), and an electronic adaptation of the Multiple Sclerosis Functional Composite (23). The adapted Multiple Sclerosis Functional Composite includes a processing speed test that is similar to the Symbol Digit Modalities Test; a manual dexterity test, similar to the 9-Hole Peg Test; a contrast sensitivity test, similar to the Sloan low-contrast visual acuity test; and a 25-foot walking speed test, similar to the Timed 25-Foot Walk. The MSPT is administered during routine clinical visits, typically prior to meeting with the healthcare provider. Depending on the institution, the results are immediately available to the healthcare provider via the patient's electronic medical record or via a results screen on the MSPT. The processing speed test, manual dexterity test, contrast sensitivity test, and 25-foot walking speed test demonstrate reliability, validity, and sensitivity to MS outcomes (13, 14).

In MS PATHS, the participating healthcare institutions collaborate with Biogen and Siemens Healthineers to implement two highly standardized MRI acquisition sequences (a three-dimensional magnetization-prepared rapid gradient-echo imaging and three-dimensional fluid-attenuated inversion recovery) that are readily available product sequences and consistent with recent MS imaging guidelines (24). The participating healthcare institutions have also implemented standardized fields in the radiology report for assessment of new or enlarging T2 lesions and contrast-enhancing lesions (if applicable).

Health Information Exchange Architecture

MS PATHS is enabled by health IT that supports the secure transfer, processing, and harmonization of clinical data for research purposes. Similar to the hub-and-spoke model of data transfer in health information exchanges, clinical source systems such as the electronic medical record systems at participating healthcare institutions (the spokes) will send data through a series of intermediary systems before the data are passed through to the central MS PATHS research database (also known as the LHS).

Each participating institution is supported by a separate gateway that is responsible for applying consent logic to incoming data and then conducting any needed format transformations to enable ingestion by a central data broker. The two data brokers, one for the United States and one for the European Union, manage a patient registration and consent index as well as a de-identification tool that pseudonymizes data. Patient data other than consent and registration information are deleted from each broker after five business days.

Biogen has contracted with an IT vendor to act as a trusted third party to build and operate the MS PATHS gateways and data brokers. This IT vendor is responsible for protecting and processing identifiable patient data before they are pseudonymized and sent to the LHS.

In the United States, the vendor has entered into business associate agreements with each participating healthcare institution. Biogen does not have access to the data in either broker. In the European Union, each healthcare institution has contracted with an additional IT vendor to serve as the initial intermediary prior to the edge gateway. The EU IT vendor completes an initial pseudonymization of the data before they are transferred to an institution's gateway and subsequently to the EU broker. Once the data are pseudonymized in a broker, they are transferred to the LHS. The LHS is logically isolated from the brokers so that both Biogen and researchers from participating healthcare institutions do not have access to identifiable patient data.

In the LHS, data are harmonized using industry-accepted clinical terminology standards such as SNOMED, Logical Observation Identifiers Names and Codes, RxNorm, National Drug Code, and International Statistical Classification of Diseases and Related Health Problems. The LHS data are available to be requested for research purposes by any researcher at a participating healthcare institution or at Biogen.

Statistical Analysis

Descriptive analyses were used to describe MS PATHS patient characteristics at the time of a patient's initial MSPT assessment. Continuous variables were reported as mean (SD) and categorical variables as percentages. Between-group differences were assessed using t tests and chi-square (χ²) tests, as appropriate.

To test whether MS PATHS is more diverse than typical phase 3 clinical trials in MS, the characteristics of patients with relapsing–remitting MS (RRMS) enrolled in MS PATHS were compared with patients with RRMS pooled from six phase 3 randomized controlled clinical trials (RCTs) sponsored by Biogen (25–29). Comparisons of patients from MS PATHS vs. the pooled RCTs were made using Wilcoxon rank-sum tests and Pearson χ² tests. Standardized mean or proportion differences (i.e., effect sizes) were also calculated. C statistics for membership (30) and predicted probability (i.e., propensity score) distributions were estimated using multivariable logistic regression models to assess overall baseline characteristics (i.e., case–mix) similarity. To assess the extent of overlap in the two populations (MS PATHS vs. RCTs), 1:1 propensity score matching (31) was performed based on a 5:1 greedy match algorithm (32). Comparisons between the matched sample characteristics were made using McNemar's test or Wilcoxon signed rank test. Effect sizes and C statistics for membership were also reported.

To assess heterogeneity in terms of assessment frequency, separate Kaplan–Meier plots were created for each center, describing the time (months) between assessments. Heterogeneity at the level of the healthcare institution was tested using log-rank tests. Separate analyses were conducted for MSPT and MRI assessments.

Results

MS PATHS Patient Population

As of August 5, 2019, 16,568 patients from 10 participating institutions in the United States (n = 7) and the European Union (n = 3) agreed to share their clinical data, representing 71.4% of patient census seen in the MS clinics within the network. As of the cutoff date for the data included in this manuscript, the number of withdrawals was 158 (0.95%) and ranged from 0.12 to 2.91% of participants across the sites.

Of 16,568 enrolled patients, 14,643 patients completed at least one MSPT assessment; the characteristics at the time of initial MSPT assessment for these patients are shown in Table 4. Mean (SD) age was 47.0 (12.4) years and the population was largely female [n = 10,712 (73.2%)] and predominantly white, although the absolute number of non-white participants is substantial. Mean (SD) Neuro-QoL T-scores ranged from 45.4 (9.6) to 52.7 (10.2).

TABLE 4

Table 4. Patient demographic and clinical characteristics at initial assessment.

At their initial MSPT assessment, 69.6% of 14,643 patients reported the use of a DMT. The most frequently reported DMTs were dimethyl fumarate, glatiramer acetate, and fingolimod. DMT use was highest in patients with RRMS [6,495 (74.6%)] compared with patients with progressive MS [1,662 (66.4%) secondary progressive, 540 (49.1%) primary progressive, and 806 (64.6%) progressive relapsing; χ² = 349.5; p < 0.0001].

The neuroperformance scores at initial assessment are summarized in Table 5. Patients with RRMS performed better on neuroperformance testing compared with patients with progressive disease.

TABLE 5

Table 5. Neuroperformance scores at initial assessment.

Assessment Completion Rates and Data Volume

The initial completion rates for the MSPT component modules ranged from 56.5% (8,277/14,643; contrast sensitivity test) to 100.0% (14,643/14,643; MyHealth module; Table 6). Missing data resulted from the coordinator disabling the module for patients unable or unwilling to complete the test or the patient canceling the test themselves. Longitudinal MSPT data were available for 72.7% (10,640/14,643) of patients (Table 7). The average patient contributed 15.6 person-months of follow-up (95% CI: 15.5–15.8); overall, 166,158 person-months of follow-up have been accumulated.

TABLE 6

Table 6. MSPT module completion rates at initial assessment.

TABLE 7

Table 7. MS PATHS data volume (N = 16,568).

A total of 14,414 MRI studies were collected from 8,364 unique patients, including 3,822/8,364 (45.7%) patients with longitudinal MRI data (Table 7). Of the 14,414 MRI studies received, 281 (2.0%) were rejected for being incomplete or not acquired using standardized sequence parameters. Of 14,643 patients with at least one MSPT assessment, 7,622 (52.1%) had at least one standardized 3T MRI.

A one-time genetic sample was collected from 6,320 patients. Other blood samples have been collected at a total of 10,223 biobanking visits from 6,581 unique patients, including 2,584/6,581 (39.3%) with a longitudinal sample collected (Table 7). The 10,223 biobanking visits represent 129,986 individual samples received in the central laboratory. A total of 265 (0.2%) samples were rejected after the initial quality control checks.

Heterogeneity Observed Across Sites

After completing their initial MSPT, 64% and 81% of patients in MS PATHS completed a follow-up MSPT within 12 and 24 months, respectively (Figure 1A). The median time to complete a follow-up MSPT was ~7.4 months after the initial MSPT. The median time to complete a follow-up MSPT varied considerably among the participating healthcare institutions, ranging from 3 to 13 months (Figure 1B; χ² = 2,575; p < 0.0001).

FIGURE 1

Figure 1. Probability of completing a follow-up Multiple Sclerosis Performance Test (MSPT; as a function of months between the initial MSPT and completing the second MSPT), (A) overall and by (B) Multiple Sclerosis Partners Advancing Technology and Health Solutions center.

After completing their initial standardized 3T brain MRI, 30% and 58% of patients in MS PATHS completed a follow-up standardized 3T brain MRI within 12 and 24 months, respectively (Figure 2A). The median time between the two standardized 3T brain MRIs was ~18 months. The median time to complete a follow-up standardized 3T brain MRI varied considerably among the participating healthcare institutions, ranging from 12 to 24 months (Figure 2B; n = 721; p < 0.0001). In four institutions, half of the patients had not yet had a second standardized 3T brain MRI.

FIGURE 2

Figure 2. Probability of completing a follow-up standardized magnetic resonance imaging on a Siemens 3T scanner, (A) overall and (B) by individual Multiple Sclerosis Partners Advancing Technology and Health Solutions center.

DMT use varied across the participating sites, in terms of both whether or not a patient was treated with a DMT (χ² = 163.3; p < 0.0001) and which DMT was prescribed (χ² = 2,584.6; p < 0.0001; Table 8).

TABLE 8

Table 8. Distribution of DMT use by participating healthcare institutions at initial assessment^*.

Heterogeneity Relative to Phase 3 MS RCTs

The patients in MS PATHS were significantly older and had a longer disease duration than the patients from the pooled RCTs (Table 9). In addition, the proportion of patients with no relapses in the last 12 months was higher in MS PATHS (54.4%) than in the pooled RCTs (1.4%). The logistic regression models predicting membership in MS PATHS resulted in a C statistic of 0.91 vs. the pooled RCTs. A C statistic of 1.0 indicates no overlap; 0.91 indicates that RCT patients were substantially distinct from patients in MS PATHS. Propensity score 1:1 matching of the 8,708 patients with RRMS in MS PATHS and 6,574 patients with RRMS in the pooled RCTs yielded 1,922 patients from each sample that were able to be matched (Table 9).

TABLE 9

Table 9. Demographic and clinical characteristics of patients with RRMS in MS PATHS and Biogen phase 3 RCTs (25–29).

Research Utilization of LHS Data

The governing subcommittee has approved 58 data use requests and one sample use request. Based on these requests, 78 conference abstracts and manuscripts have been published. Two National Institutes of Health grants have been funded.

Discussion

The Institute of Medicine, a member of the US National Academies, recommended foundational elements for an LHS (10) as an approach to address recognized challenges, including the need for consistent quality and efficiency, in the US healthcare system. These challenges are also recognized by healthcare systems and regulatory bodies outside of the United States such as the European Medicines Agency (2, 4, 33–35). As the first LHS in MS, MS PATHS was designed to incorporate each of the foundational elements outlined in the introduction.

There were three initial observations from MS PATHS. The first relates to the characteristics of patients with MS. Although the patients in MS PATHS demonstrate demographic and disease characteristics typical of other large MS cohorts, the clinical characteristics from patients with RRMS in MS PATHS are partially non-overlapping with the patients enrolled in clinical trials (25–29), thus reflecting a broader population. The patients in MS PATHS exhibited a broader range of ages, disability levels, disease duration, and DMT use, presumably because MS PATHS aims to enroll all patients with a diagnosis of MS, including clinically isolated syndrome, with no additional inclusion or exclusion criteria (e.g., requiring a relapse in the past 12 months, as is typically required in pivotal RCTs). MS PATHS is more racially diverse than previously reported trials; white patients make up 79.5% of the US MS PATHS population in comparison with 93.7% of the oral DMT clinical trial study population (36). The minimal inclusion criteria and the robust enrollment rates suggest that studies using the MS PATHS population will generate data that will be more generalizable than data from clinical trials, in which enrollment is more selective. It is also likely that patient characteristics obtained from MS PATHS will be more representative of the broader MS population than data from registries where a smaller subset of patients from a clinical practice in one location is selected for inclusion in the cohort (37).

The second observation is the demonstrated feasibility of real-time quantitative patient phenotyping as part of clinical practice. As of August 5, 2019, 88.4% of patients who gave permission for their clinical data to be used for research have completed at least one self-administered neuroperformance assessment. Also, standardized MRI acquisition sequences were incorporated into clinical brain MRI protocols, with 98% of scans passing the quality control assessments.

The third observation relates to the variability in practice patterns observed across participating healthcare institutions. We observed a significant inter-institution variability in the rate of return for follow-up and the average interval between assessments captured in MS PATHS, indicating that there is no uniform standard for visit frequency in the network. Follow-up interval is important when determining outcomes because visit frequency may influence observed event frequency, depending on how the information is captured. For example, individuals with MS may not always present to the physician during a relapse (38).

MS PATHS has experienced challenges and learnings during the initial implementation, including:

(1) Creating trust in an academic–industry collaboration: The initial collaborators were excited about the vision of MS PATHS; however, there were questions on how to build trust in a multi-party collaboration with a biopharmaceutical company. The initial collaborators recognized that a transparent governance process would be key to engender trust among the participants and in the scientific community at large. All parties quickly aligned on a vision of using the LHS concept to generate evidence to improve outcomes for patients with MS as well as the guiding principles (Table 1). The governance model was then developed over the course of six all-day meetings to enable all parties to fulfill the shared vision and operate by the guiding principles.

(2) Aligning on which data to collect and how to standardize data collection: Given the broad research goals of MS PATHS, the potential data to collect were vast. In the initial planning discussions, it became clear that standardizing and collecting all potential data would not fit into the clinical workflow. Data were prioritized in terms of time and utility. The goal was to keep MSPT administration no longer than 30 minutes for the average patient with MS, keep the brain MRI total scan time unchanged, and add no additional time to a clinician's clinical documentation. Any suggested patient-reported question or scale was weighed in terms of time to complete and its potential utility for clinical care and research. Implementing axial reconstruction of the standardized three-dimensional sequences allowed radiologists/neurologists to view scans as they normally would without the need for duplicative MRI acquisitions. Also, collaborating with Siemens Healthineers allowed the sequences to be optimized based on feedback from MS PATHS radiologists to enable broad adoption in the network.

(3) Implementation of common health IT platforms: Generating data in real time for clinical decision-making and providing pseudonymized data for research involved major technical challenges. We started with a basic design of a health information exchange platform and iterated after receiving feedback from key IT stakeholders at the founding institutions. Because IT architecture is different in each medical center, interfaces for the IT platforms need to be customized for each medical center to ensure compatibility and adherence to security requirements. The current implementation maintains consistent data handling and processing across the network but flexes to allow for variability in the interfaces and whether a push or pull model is optimal for each individual medical center. This type of model was aided by the increasing adoption of common healthcare data standards across healthcare in general.

(4) Ensuring compliance with US and EU data privacy regulations: MS PATHS was designed to be both HIPAA and GDPR compliant. In MS PATHS, patient data are only shared for research purposes, with patient consent. Trusted third parties serve as intermediaries to remove patient identifiers before the data are aggregated for research under appropriate contractual arrangements. Garnering acceptance and approval from a myriad of stakeholders at each institution is equally as important as a sound program design. MS PATHS has shown that with a sound program design, stakeholder engagement, and patient permission, data aggregation is feasible to conduct under both HIPAA and GDPR.

(5) Enabling efficient data sharing: Our chosen hub-and-spoke data sharing model simplified contractual relationships in MS PATHS. Each healthcare institution negotiated directly with Biogen rather than having to enter into a multi-lateral negotiation for the initial contract and any subsequent amendments. All stakeholders acknowledged the logistical benefits of this model. However, it required key clauses on the scope of data sharing, data permissions, and intellectual property to be uniform across all contracts and informed consent forms and for all parties to be comfortable that we could not accommodate one-off deviations if we were to maintain an effective hub-and-spoke model.

(6) Re-engineering of clinical workflows involved all aspects of clinic operations: One example of how MS clinics needed to change their operations was the incorporation of the MSPT into the clinical workflow. This necessitated designating space in the clinic as a MSPT testing area, training staff on the MSPT, introducing MS patients to a new aspect of their visit, and adjusting patient arrival times whether formally through a new appointment time or reminders to arrive 30 min before their scheduled visit with their healthcare provider. The workflow challenges differed somewhat for implementing the MSPT, implementing the standardized MRI protocol, and research workflows such as biobanking. Enthusiasm for the LHS tenets and the project sustained commitment from academic and industry project leaders, overcoming a myriad of challenges. The network investigators also committed to sharing best practices through conference calls, investigator meetings, and site visits. For example, an early MSPT implementation insight shared across the network was that patients generally have a better testing experience if they take the test without staff, a family member, or a caregiver by their side.

(7) Real-life experience highlighted gaps in original technical assumptions: Some design issues that have emerged and are being corrected include issues with MSPT functionality (e.g., lower-than-expected contrast sensitivity test completion rates) and issues with optimizing data collection methods for certain variables (e.g., comorbidities, medication start/stop dates). Among the advantages of an academic and industry partnership model are the real-time and continuous feedback, re-assessment, and refinement of data gathering and overall strategy that facilitate continuous improvement. One example of this feedback loop is the creation of a MS SmartForm by one of the MS PATHS institutions that is freely available in EPIC foundation to all EPIC users. This SmartForm will facilitate the standardization of key MS-related variables such as relapses and disease-modifying therapy start/stop dates to primarily aid clinical care, but with the secondary benefit of improving data quality for research. The US MS PATHS centers are currently in various stages of SmartForm implementation, while we are coordinating with the EU MS PATHS centers to harmonize as much as possible.

(8) Incorporating the voice of the patient: Patients have always been identified as a key stakeholder in MS PATHS. During the design phase, the investigators engaged a local patient advisory group, when available, on the MS PATHS concept. The MSPT was designed to facilitate the voice of the patient being incorporated into the clinical visit through the neuroperformance modules and 12 domains of quality-of-life as assessed by the Neuro-QoL. Prior to the initial deployment of the MSPT in MS PATHS, qualitative feedback was obtained by the patients in an initial usability study (11). However, there is still more to be done to engage patients and complete the feedback loop as envisioned in a LHS (10). To start, the network is working to generate regular patient newsletters that provide updates on MS PATHS and summarize recent research presented at conferences or published in journals. Each MS center will provide these updates via their routine patient communication channels. We are also working on an updated publicly facing website that would also have this information.

(9) Moving from clinical implementation to clinical decision-making: Our efforts to date have been focused on the initial implementation and data collection as highlighted in this paper. The network is now shifting focus to evidence generation that will enable the data collected in MS PATHS to be used more routinely for clinical decision-making. Key next topics include understanding clinical cutoffs for the MSPT and MRI metrics, looking at practice heterogeneity across the network to identify best practices, and exploring feedback loops for investigators to more easily use the data for quality improvement.

Despite the promise of this research, there are limitations to be considered. Data missingness may be non-random. For example, patients with more disability may be less likely to have an MRI or complete the MSPT. The participating centers are referral centers and may not fully represent broader MS populations. For example, the rates of patients with progressive disease on a DMT may be higher than expected, and incorporation of the SmartForm data may provide additional insight. Still the MS PATHS population represents an improvement over the populations included in clinical trials as it is more racially and clinically diverse.

Another limitation is related to the restriction of standardized MRI studies to Siemens 3T scanners. This was intentional in order to remove scanner and manufacturer variability from the derived metrics as the initial step in delivering point-of-care MRI-based metrics to the clinician. As such, the time to follow-up MRI does not take into account the MRI exams acquired within each institution on non-Siemens 3T MRI scanners or using different (non-standardized) acquisition sequences or an MRI obtained outside the healthcare institution. Therefore, differences in MRI follow-up time between centers may reflect variations in the use of Siemens 3T scanners and variation in the percentage of patients who obtain MRI exams outside of the healthcare institution. A few participating centers image their patients with MS exclusively on Siemens 3T scanners, while others use a wide variety of scanner types across internal and external imaging facilities. Translation of the imaging methods to non-Siemens 3T scanners is planned.

The initial implementation of MS PATHS has been successful overall, although we are continuously optimizing aspects of our operations. We anticipate its value to grow with time as large-scale, longitudinal, and multidimensional quantitative data accumulate. These data will potentially drive new insights and aid in the development and the validation of new technologies. In particular, incorporating point-of-care quantitative MRI-based metrics in clinical practice has the potential to transform precision medicine to the same degree that MRI-based metrics have advanced clinical trials.

Another important direction for future work in MS PATHS is defining individual patient outcomes using quantitative standardized data. As MS PATHS data accumulate, it may be possible to define complete MS disease control or actionable change based on quantitative monitoring. Further, in the future, decision support based on advanced analytic methods could help neurologists care for individuals with MS in the context of a large number of DMTs and other therapeutic choices. Data accumulated in MS PATHS could support the development of predictive models and decision support systems.

The LHS concept, exemplified by MS PATHS, could accelerate translational research in MS as evidenced by the number of active proposals and grant approvals. Tens of thousands of patients may be required to unravel the biological basis for MS heterogeneity and develop personalized medicine based on individual patient characteristics. Using quantitative data derived from clinical, imaging, and laboratory assessments generated during patient care could drastically reduce the cost of, and thus make feasible, such translational research studies.

Data Availability Statement

The datasets presented in this article are not readily available because MS PATHS is an ongoing study. Requests to access the datasets should be directed to datasharing@biogen.com.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics committees or institutional review boards at the following participating centers: Cleveland Clinic, Cleveland, OH, USA; Johns Hopkins University, Baltimore, MD, USA; New York University, New York, NY, USA; OhioHealth, Columbus, OH, USA; Washington University in St. Louis, St. Louis, MO, USA; University of Rochester Medical Center, Rochester, NY, USA; Cleveland Clinic Lou Ruvo Center for Brain Health, Las Vegas, NV, USA; University Hospital of Giessen and Marburg, Marburg, Germany; Vall d'Hebron University Hospital, Barcelona, Spain; and University Hospital Carl Gustav Carus, Dresden, German. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

JW, CM, EF, and RR conceived and designed the data plan. CM and FP conducted the analyses. EM, RB, JW, and RR led the writing of the manuscript. RR had final responsibility for the decision to submit for publication. All authors provided critical feedback and helped shape the research, analysis, manuscript, had full access to the study data for interpretation and drafting of the report, and contributed to the interpretation of the findings. Editorial support was provided to authors and funded by the study funder. All decisions relating to manuscript writing and content were made jointly by the authors.

Funding

Biogen provides funding for MS PATHS. As the sponsor of MS PATHS, Biogen facilitated the planning process and funded the development and ongoing operation/support of the MSPT and health information exchange architecture. Biogen entered into a research agreement with Siemens Healthineers to co-develop the MRI metrics prototype. Biogen also compensates participating healthcare institutions for data and samples contributed to MS PATHS at fair market value rates. Biogen serves as the hub for contracting and data sharing. All the participating healthcare institutions contracted with Biogen to receive consented patient data and allow Biogen to share the data with other researchers for uses consistent with the informed consent. As the data aggregator, Biogen curates the data and samples and then makes them available to researchers in the network who have an approved data or sample use proposal. To maintain the scientific integrity of the network, Biogen plays no role in adjudicating data and sample requests from academic collaborators or in conducting any scientific or editorial review of resultant publications. Biogen also does not permit its commercial employees to access the data.

Conflict of Interest

The participating healthcare institution of each investigator receives financial compensation for data shared and biosamples collected as part of this program based on fair market value. EM reports research support from Biogen (MS PATHS and investigator-initiated studies) and Sanofi-Genzyme for investigator-initiated studies, serving as site principal investigator for a Sun Pharma study, free medication from Teva Neuroscience for use in a clinical trial, for which she is principal investigator, and royalties for editorial duties from UpToDate. RB reports consulting fees from Biogen, Genentech/Roche, Novartis, and Sanofi-Genzyme, research support from Biogen, and personal and institutional equity ownership related to the MSPT licensed to Biogen and Qr8 by Cleveland Clinic. JW, CM, EF, BK, FP, and RR are employees of and hold stock/stock options in Biogen. TB reports research support from Avid/Lilly and Roche, travel support from Biogen, and serving as clinical trial investigator for Avid/Lilly, Biogen, and Roche. CH reports speaking and consulting fees from Biogen, EMD Serono, Genentech, Genzyme, and Novartis, research support from Biogen, Genentech, and Patient-Centered Outcomes Research Institute, and research support to her institution, Cleveland Clinic, from Biogen. MH reports research support from Biogen, Chugai, National Institutes of Health (NeuroNEXT), Novartis, and Patient-Centered Outcomes Research Institute. II reports funding from Biogen in the form of a contract with her employer, Johns Hopkins University (JHU), to support effort on this project, serving as principal investigator on a research grant from Siemens paid to JHU for an unrelated project, and consultant fees from Alexion. SJ reports speaker fees from St. Jude Hospital, travel support from Siemens, grant support from Biogen for MS PATHS, and grant support from St. Jude Hospital. HK reports research support from Novartis and traveling and speaking/consulting fees and/or honoraria from Novartis, Biogen, Teva, IXICO, and Siemens. LK reports consulting fees from Biogen, Gerson Lehrman, Novartis, and Sanofi, grant support from Biogen, and receipt of royalties for the Fatigue Severity Scale licensed to various biopharmaceutical entities. YL declares no competing interests. XM reports speaker fees/travel expenses for scientific meetings or steering committees/advisory boards for clinical trials for Actelion, Biogen, Celgene, EXCEMED, Genzyme, Merck Serono, MS International Foundation, National MS Society, Novartis, Sanofi-Genzyme, Roche, and Teva. RN reports consulting fees and/or honoraria from Alexion, Alkermes, Biogen, Celgene, EMD Serono, Genentech, Genzyme, NervGen, Novartis, TG Therapeutics, and Third Rock Ventures. JN reports consulting/speaking honoraria from Biogen, EMD Serono, Genentech, Genzyme, and Novartis. AR serves on scientific advisory boards for Bayer, Biogen, Icometrix, Olea Medical, and SyntheticMR, is a consultant/speaker for Novartis, Roche, and Sanofi-Genzyme, and receives research support from Biogen. MS reports research funding for the conduct of MS PATHS from Biogen. BT reports advisory boards for Bayer, Biogen, CSL Behring, Genzyme, Grifols, Novartis, and UCB, speaker honoraria from Bayer, Biogen, CSL Behring, Genzyme, Grifols, Novartis, and Octapharma, consulting fees from Bayer, Biogen, CSL Behring, Genzyme, Grifols, Novartis, and UCB, and research support from Biogen and Novartis. MTin reports consulting/speaking honoraria from Almirall, Bayer Schering, Biogen, Genzyme, Merck Serono, Novartis, Roche, Sanofi, and Teva, grants from Biogen, Genzyme, and Novartis, and serving as co-editor of Multiple Sclerosis Journal – Experimental, Translational, and Clinical. MTiv reports funding from Biogen in the form of a contract with her employer, University of Rochester, to support efforts on MS PATHS. TZ reports speaker fees/travel expenses for scientific meetings or steering committees/advisory boards for clinical trials for Almirall, Bayer, Biogen, Celgene, Genzyme, Gilead, Merck Serono, Novartis, Roche, and Teva and grant support from BAT, Biogen, Genzyme, Merck Serono, Novartis, and Roche.

The handling editor declared a past co-authorship with AR.

Acknowledgments

This study was funded by Biogen (Cambridge, MA, USA). Biogen provided funding for medical writing and editorial support in the development of this manuscript. Linda Wagner (Excel Scientific Solutions, Fairfield, CT, USA) wrote the first draft of the manuscript based on input from the authors, and Elizabeth Wassmer (Excel Scientific Solutions, Fairfield, CT, USA) copyedited and styled the manuscript per journal requirements. Biogen reviewed and provided feedback on the paper to the authors. The authors had full editorial control of the paper and provided their final approval of all content. The authors thank all of the patients who are participating in the MS PATHS initiative.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur.2020.00632/full#supplementary-material

Abbreviations

DMT, disease-modifying therapy; EMR, electronic medical record; FLAIR, fluid-attenuated inversion recovery; IQR, interquartile range; IT, information technology; LHS, learning health system; MPRAGE, magnetization-prepared rapid gradient-echo imaging; MRI, magnetic resonance imaging; MS, multiple sclerosis; MS PATHS, Multiple Sclerosis Partners Advancing Technology and Health Solutions; MSPT, Multiple Sclerosis Performance Test; NA, not applicable; Neuro-QoL, Quality of Life in Neurological Disorders; PPMS, primary progressive multiple sclerosis; PRMS, progressive relapsing multiple sclerosis; RCT, randomized controlled clinical trial; RRMS, relapsing-remitting multiple sclerosis; SPMS, secondary progressive multiple sclerosis.

References

1. Rotstein D, Montalban X. Reaching an evidence-based prognosis for personalized treatment of multiple sclerosis. Nat Rev Neurol. (2019) 15:287–300. doi: 10.1038/s41582-019-0170-8

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Gourraud PA, Henry RG, Cree BA, Crane JC, Lizee A, Olson MP, et al. Precision medicine in chronic disease management: the multiple sclerosis BioScreen. Ann Neurol. (2014) 76:633–42. doi: 10.1002/ana.24282

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Frieden TR. Evidence for health decision making — beyond randomized, controlled trials. N Engl J Med. (2017) 377:465–75. doi: 10.1056/NEJMra1614394

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, et al. Real-world evidence — what is it and what can it tell us? N Engl J Med. (2016) 375:2293–7. doi: 10.1056/NEJMsb1609216

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Ziemssen T, Hillert J, Butzkueven H. The importance of collecting structured clinical information on multiple sclerosis. BMC Med. (2016) 14:81. doi: 10.1186/s12916-016-0627-1

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Confavreux C. Establishment and use of multiple sclerosis registers–EDMUS. Ann Neurol. (1994) 36:S136–9. doi: 10.1002/ana.410360733

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Paty D, Studney D, Redekop K, Lublin F. MS COSTAR: a computerized patient record adapted for clinical research purposes. Ann Neurol. (1994) 36:S134–5. doi: 10.1002/ana.410360732

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Cohen JA, Trojano M, Mowry EM, Uitdehaag BM, Reingold SC, Marrie RA. Leveraging real-world data to investigate multiple sclerosis disease behavior, prognosis, and treatment. Mult Scler. (2020) 26:23–37. doi: 10.1177/1352458519892555

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Flachenecker P, Buckow K, Pugliatti M, Kes VB, Battaglia MA, Boyko A, et al. Multiple sclerosis registries in Europe - results of a systematic survey. Mult Scler. (2014) 20:1523–32. doi: 10.1177/1352458514528760

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Institute of Medicine. Best Care at Lower Cost. The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press. (2013).

Google Scholar

11. Rhodes JK, Schindler D, Rao SM, Venegas F, Bruzik ET, Gabel W, et al. Multiple sclerosis performance test: technical development and usability. Adv Ther. (2019) 36:1741–55. doi: 10.1007/s12325-019-00958-x

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Rudick RA, Miller D, Bethoux F, Rao SM, Lee JC, Stough D, et al. The Multiple Sclerosis Performance Test (MSPT): an iPad-based disability assessment tool. J Vis Exp. (2014) 88:e51318. doi: 10.3791/51318

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Rao SM, Losinski G, Mourany L, Schindler D, Mamone B, Reece C, et al. Processing speed test: validation of a self-administered, iPad®-based tool for screening cognitive dysfunction in a clinic setting. Mult Scler. (2017) 23:1929–37. doi: 10.1177/1352458516688955

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Rao SM, Galioto R, Sokolowski M, McGinley M, Freiburger J, Weber M, et al. Multiple Sclerosis Performance Test: validation of self-administered neuroperformance modules. Eur J Neurol. (2020) 27:878–86. doi: 10.1111/ene.14162

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Nickerson M, Cofield SS, Tyry T, Salter AR, Cutter GR, Marrie RA. Impact of multiple sclerosis relapse: the NARCOMS participant perspective. Mult Scler Relat Disord. (2015) 4:234–40. doi: 10.1016/j.msard.2015.03.005

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Nickerson M, Marrie RA. The multiple sclerosis relapse experience: patient-reported outcomes from the North American Research Committee on Multiple Sclerosis (NARCOMS) registry. BMC Neurol. (2013) 13:119. doi: 10.1186/1471-2377-13-119

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Hohol MJ, Orav EJ, Weiner HL. Disease steps in multiple sclerosis: a longitudinal study comparing disease steps and EDSS to evaluate disease progression. Mult Scler. (1999) 5:349–54. doi: 10.1177/135245859900500508

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Kister I, Chamot E, Salter AR, Cutter GR, Bacon TE, Herbert J. Disability in multiple sclerosis: a reference for patients and clinicians. Neurology. (2013) 80:1018–24. doi: 10.1212/WNL.0b013e3182872855

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Learmonth YC, Motl RW, Sandroff BM, Pula JH, Cadavid D. Validation of Patient Determined Disease Steps (PDDS) scale scores in persons with multiple sclerosis. BMC Neurol. (2013) 13:37. doi: 10.1186/1471-2377-13-37

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Gershon RC, Lai JS, Bode R, Choi S, Moy C, Bleck T, et al. Neuro-QoL: quality of life item banks for adults with neurological disorders: item development and calibrations based upon clinical and general population testing. Qual Life Res. (2012) 21:475–86. doi: 10.1007/s11136-011-9958-8

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Miller DM, Bethoux F, Victorson D, Nowinski CJ, Buono S, Lai JS, et al. Validating neuro-QoL short forms and targeted scales with people who have multiple sclerosis. Mult Scler. (2016) 22:830–41. doi: 10.1177/1352458515599450

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Cella D, Lai JS, Nowinski CJ, Victorson D, Peterman A, Miller D, et al. Neuro-QoL: brief measures of health-related quality of life for clinical research in neurology. Neurology. (2012) 78:1860–7. doi: 10.1212/WNL.0b013e318258f744

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Rudick RA, Cutter G, Reingold S. The multiple sclerosis functional composite: a new clinical outcome measure for multiple sclerosis trials. Mult Scler. (2002) 8:359–65. doi: 10.1191/1352458502ms845oa

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Traboulsee A, Simon JH, Stone L, Fisher E, Jones DE, Malhotra A, et al. Revised recommendations of the Consortium of MS Centers task force for a standardized MRI protocol and clinical guidelines for the diagnosis and follow-up of multiple sclerosis. AJNR Am J Neuroradiol. (2016) 37:394–401. doi: 10.3174/ajnr.A4539

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Calabresi PA, Kieseier BC, Arnold DL, Balcer LJ, Boyko A, Pelletier J, et al. Pegylated interferon beta-1a for relapsing-remitting multiple sclerosis (ADVANCE): a randomised, phase 3, double-blind study. Lancet Neurol. (2014) 13:657–65. doi: 10.1016/S1474-4422(14)70068-7

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Fox RJ, Miller DH, Phillips JT, Hutchinson M, Havrdova E, Kita M, et al. Placebo-controlled phase 3 study of oral BG-12 or glatiramer in multiple sclerosis. N Engl J Med. (2012) 367:1087–97. doi: 10.1056/NEJMoa1206328

CrossRef Full Text | Google Scholar

27. Gold R, Kappos L, Arnold DL, Bar-Or A, Giovannoni G, Selmaj K. Placebo-controlled phase 3 study of oral BG-12 for relapsing multiple sclerosis. N Engl J Med. (2012) 367:1098–107. doi: 10.1056/NEJMoa1114287

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Polman CH, O'Connor PW, Havrdova E, Hutchinson M, Kappos L, Miller DH, et al. A randomized, placebo-controlled trial of natalizumab for relapsing multiple sclerosis. N Engl J Med. (2006) 354:899–910. doi: 10.1056/NEJMoa044397

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Rudick RA, Stuart WH, Calabresi PA, Confavreux C, Galetta SL, Radue EW, et al. Natalizumab plus interferon beta-1a for relapsing multiple sclerosis. N Engl J Med. (2006) 354:911–23. doi: 10.1056/NEJMoa044396

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KGM. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. (2015) 68:279–89. doi: 10.1016/j.jclinepi.2014.06.018

PubMed Abstract | CrossRef Full Text | Google Scholar

31. D'Agostino RB Jr. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med. (1998) 17:2265–81. doi: 10.1002/(SICI)1097-02581998101517:19<2265::AID-SIM918>3.0.CO

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Parsons LS. Reducing bias in a propensity score matched pair sample using greedy matching techniques. In: Proceedings of the Twenty-Sixth Annual SAS Users Group International Conference. Cary, NC: SAS Institute (2004).

Google Scholar

33. European Medicines Agency. Report on Multiple Sclerosis Registries - Workshop 7 July 2017: Patient Registry Initiative. (2017) Available online at: https://www.ema.europa.eu/en/documents/report/report-multiple-sclerosis-registries_en.pdf (accessed May 3, 2019).

Google Scholar

34. European Medicines Agency. Patient Registries Workshop, 28 October 2016: Observations and Recommendations Arising From The Workshop. (2017) Available online at: https://www.ema.europa.eu/en/documents/report/report-patient-registries-workshop_en.pdf (accessed May 3, 2019).

Google Scholar

35. European Medicines Agency. EMA Regulatory Science To 2025: Strategic Reflection. (2018) Available online at: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/ema-regulatory-science-2025-strategic-reflection_en.pdf (accessed May 3, 2019).

Google Scholar

36. Avasarala J. Inadequacy of clinical trial designs and data to control for the confounding impact of race/ethnicity in response to treatment in multiple sclerosis. JAMA Neurol. (2014) 71:943–4. doi: 10.1001/jamaneurol.2014.79

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Ziemssen T, Kern R, Thomas K. Multiple sclerosis: clinical profiling and data collection as prerequisite for personalized medicine approach. BMC Neurol. (2016) 16:124. doi: 10.1186/s12883-016-0639-7

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Nazareth TA, Rava AR, Polyakov JL, Banfe EN, Waltrip II RW, Zerkowski KB, et al. Relapse prevalence, symptoms, and health care engagement: patient insights from the multiple sclerosis in America 2017 survey. Mult Scler Relat Disord. (2018) 26:219–34. doi: 10.1016/j.msard.2018.09.002

PubMed Abstract | CrossRef Full Text | Google Scholar