Adaptation for Regulatory Application: A Content Analysis of FDA Risk Evaluation and Mitigation Strategies Assessment Plans (2014–2018) Using RE-AIM

Background: Risk Evaluation and Mitigation Strategies (REMS) are safety programs that U.S. Food and Drug Administration can require to ensure a drug's benefits outweigh its risks and can be considered public health interventions. FDA's 2019 draft Guidance for Industry on REMS Assessments encourages the development of “novel methods for assessing REMS [to] help advance the science of post-market assessment of effectiveness of risk mitigation strategies.”

Objective: To characterize REMS assessment plans using RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework and identify areas for advancing methods for evaluating REMS programs. RE-AIM was selected for its wide application evaluating the translation of scientific advances into practice for public health impact.

Methods: A content analysis of REMS assessment plans (N = 18) and measures(n = 540) was conducted for REMS programs approved by FDA between 1/1/2014–12/31/2018. Eligibility criteria were: a new drug application or biologic license application, included FDA-mandated mitigation strategies called elements to assure safe use (ETASU), and represented a single product REMS program. Assessment plans were collected from publicly available regulatory approval letters from REMS@FDA website. Blinded reviewers categorized each REMS assessment measure to a RE-AIM dimension, adjudicated their application (average IRR 75%), and refined the adapted dimensions' definitions. Dimensions were also mapped to REMS Assessment guidance categories.

Results: The median number of assessment measures per REMS assessment plan was 31 (IQR: 21–36). Frequency of measures per RE-AIM criteria per REMS program was: Reach (median = 2; IQR: 2–4); Effectiveness (median = 2.5; IQR:1–4); Adoption (median = 3.5; IQR: 2–5); Implementation (median = 18; IQR: 15–24); Maintenance (median = 0; IQR: 0–1). Adoption (among prescriber, health system agents of implementation) was more commonly assessed than Reach (population-attributable number of patients affected). Assessment of heterogeneity of Adoption and Reach was generally absent. Implementation assessment measures were most common among drugs requiring evidence of safe-use conditions before dispensing or administering the drug. Patient-level Effectiveness and Maintenance assessments were most common among drugs requiring patient monitoring.

Discussion: Implementation science frameworks, such as RE-AIM, can be applied to characterize REMS assessment measures and identify opportunities for standardizing and strengthening their evaluation. Methods to measure Maintenance are needed to provide real-world evidence of REMS integration into the healthcare system.

Introduction

The U.S. Food and Drug Administration (FDA) is responsible for protecting the public health of Americans by assuring the safety and efficacy of human drugs and biological products (1). Over the past two decades, modernization of post marketing drug safety and risk management has received increasing attention (2, 3). Post marketing safety issues include serious adverse events, product quality issues, and medication errors (4). Given the U.S. population's large and increasing magnitude of medication exposure, the potential for harms from adverse drug events constitutes a critical patient safety and public health challenge. An estimated one-third of all hospital adverse events and over 3.5 million physician office visits each year are attributable to adverse drug events (5).

The Food and Drug Administration Amendments Act (FDAAA) of 2007 granted FDA authority to require risk evaluation and mitigation strategies (REMS) to ensure that the benefits of a drug outweigh its risks (6). REMS are required risk management plans that use risk minimization strategies beyond the professional product labeling (7). REMS can be required for existing drugs on the market, new drug applications (NDAs), abbreviated NDAs (ANDAs) for generic drugs, and biologics license applications (BLAs) (6). Between enactment of FDAAA and September 2019, 284 REMS programs have been approved by FDA for a wide-range of therapeutic areas affecting the treatment of obesity and diabetes, depression, and pain management (8). Please see Table 1 for definitions of common FDA and REMS terms.

TABLE 1

Table 1. Common terms and acronyms for US Food and Drug Administration (FDA) and Risk Evaluation and Mitigation Strategies (REMS).

Early in the implementation of REMS, the majority of programs included strategies focused on dissemination of risk information. REMS programs may require that drug manufacturers develop materials for patients, such as a Medication Guide, which contain FDA-approved information in patient-friendly language that can help inform patients about how to use a medication and avoid serious adverse events. After guidance issuance in 2012, FDA no longer required every Medication Guide to be part of a REMS, however, they still remain part of the FDA-approved labeling (9). In most cases, FDA includes a Medication Guide as part of a REMS only when the REMS includes other clinical interventions such as patient counseling (10). Other dissemination strategies include targeting healthcare providers; these are known as Communication Plans. REMS may require drug manufacturers to communicate directly to healthcare providers involved in the delivery of health care or medications or develop certain packaging and safe disposal technologies (11, 12). Most of the REMS that included only a Medication Guide or Communication Plan have now been released under the mandate of REMS.

Today, the majority of active REMS programs (84%, 51 out of 61programs) involve complex multi-level interventions (8). In these situations, FDA requires healthcare providers to conduct clinical interventions known as elements to assure safe use (ETASU) that support the safe use of the medication. ETASU may include: training or certification of prescribers, training or certification of dispensers, dispensing/administering the drug in certain settings, requiring evidence or documentation of safe use conditions, monitoring of patients, and/or enrolling patients in a registry (13).

REMS programs, although developed by drug manufactures, are essentially one form of public health intervention programs that need to be implemented within the US healthcare system and adopted by healthcare providers. For example, the Opioid Analgesics REMS program is one strategy among multiple national and state efforts to reduce the risk of abuse, misuse, addiction, overdose, and deaths due to prescription opioid analgesics. It requires that training be made available to all healthcare providers who are involved in the management of patients with pain, including nurses and pharmacists (14). In 2009, the Zyprexa Relprevv REMS was approved to reduce the risk of post-injection delirium sedation syndrome. The REMS was developed to make sure all patients receive special monitoring during the period just following drug administration when post-injection delirium sedation syndrome is most likely to occur, so it can be detected and treated (15).

Drug manufacturers are also required to assess the effectiveness of their REMS program and submit assessment reports to FDA at specified frequency. Manufacturers generally develop a REMS assessment plan prior to approval. The REMS assessment plan is a specific plan for how the drug manufacturer intends to assess the performance of the REMS in meeting its risk mitigation goals and objectives (10). Each assessment plan includes a number of assessment measures to evaluate processes and outcomes. Depending on the complexity of the program, the number of assessment measures may vary. An example of a measure assessing processes may include the number of prescribers, health care settings, and pharmacies that have undergone training in the REMS program. An example of a measure assessing outcomes may include numbers and rates of a specific adverse event of interest such as rates of serious bleeds or severe neutropenia (16). The REMS assessment plan is outlined in the original REMS approval letters for all NDAs and BLAs and is made publicly available through the FDA website, REMS@FDA (also available at DRUGS@FDA).

Assessing the effectiveness of REMS programs is challenging. For example, drug manufacturers and healthcare providers have expressed concerns associated with the challenges of collecting data and lack of standardized format for assessment plans (17). Early following the implementation of REMS, the Office of the Inspector General report raised concerns about the effectiveness of the REMS programs and recommended that FDA should develop and implement a plan to identify, develop, validate, and assess REMS components (2). The report also recommended that FDA should identify and implement reliable methods to assess the effectiveness of REMS.

In response and to modernize post-marketing drug safety, the FDA committed as part of the fifth authorization of the prescription drug user fee program to develop evidence-based methodologies for assessing the effectiveness of REMS (3). The Prescription Drug User Fee Act (PDUFA) gives FDA authority to collect fees from companies that produce drugs when they submit NDA and BLA applications in exchange for ensuring timely review; PDUFA is reauthorized by Congress every 5 years, providing new windows of opportunity for advancing public policy by allowing manufacturers and the FDA to discuss and negotiate commitments to facilitate “timely access to safe, effective, and innovated new medicines for patients.”

In 2019, FDA issued a draft guidance entitled “REMS Assessments: Reporting and Planning” (henceforth referred to as the Assessment Guidance) in which it encouraged “applicants and the research community to develop novel methods for assessing REMS” (10). The draft Assessment Guidance outlines five categories for evaluation, including: Outreach and Communications, Implementation and Operations, Knowledge, Safe-Use Behaviors, and Health Outcomes; see Table 2.

TABLE 2

Table 2. Adaptation of RE-AIM dimensions as applied to REMS assessment measures and Assessment Guidance categories.

Using the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework may be suitable for evaluating REMS programs. RE-AIM is a framework to enhance the translation of research into practice through the adoption and implementation of evidence-based interventions. The framework was initially used to evaluate prevention and health behavior change programs, and more recently, has been used to help plan programs and improve their chances of working in “real-world” settings. The overall goal of the RE-AIM framework is to encourage program planners, evaluators, researchers, funders, and policymakers to consider essential program elements including external validity. Its five dimensions are designed to enhance the quality, speed, and impact of public health efforts and involve the following: reach of intended target population, effectiveness on important outcomes, adoption by target staff or settings, implementation consistency, and maintenance of intervention effects over time in individuals and settings (20).

RE-AIM addresses all components of REMS programs, including compliance processes, program participation, and overall outcomes, as suggested by the Assessment Guidance (10). Moreover, RE-AIM is an evaluation framework from implementation science that has been widely applied to evaluate health interventions similar to REMS programs (18). For example, RE-AIM has been used by the Centers for Disease Control and Prevention (CDC) for the evaluation of the implementation of the Diabetes Prevention Group (21). Another example includes the application of RE-AM to evaluation of implementing physical activity as a standard of care in healthcare settings (22). Its application has been proposed as an extension to assess the public health impact of policy change (23). The objective of this study was to characterize REMS assessment plans using RE-AIM and to identify areas for advancing methods for evaluating REMS programs.

Methods

A content analysis of REMS assessment plans (N = 18) and measures(n = 540) was conducted for REMS programs approved by the FDA between January 1, 2014 and December 31, 2018. Given that the first REMS was approved in 2008, we limited our study sample to REMS programs approved in the past 5 years as these would be more aligned with current policy. Programs were excluded if they had been released during this timeframe and were no longer required by the FDA to be implemented. We also excluded REMS containing Medication Guides and Communication Plans as the sole elements because we wanted to study complex multi-level, multi-system interventions, leaving active REMS with ETASU for analysis. Finally, shared system REMS were excluded because we wanted to focus on new programs, and shared system REMS programs reflect sustaining programs that have been adapted for generic products. Figure 1 shows the selection process.

FIGURE 1

Figure 1. Flow diagram of 2014–2018 active, single product REMS with ETASU program selection for content analysis of assessment plans using the RE-AIM framework. ANDA, Abbreviated New Drug Application; CP, Communication Plan; MG, Medication Guide; *REMS programs were accessed for eligibility January 2019.

Source Data

The assessment plan for each REMS program was obtained from the publicly-available regulatory approval letters downloaded from the FDA's website REMS@FDA on January 15, 2019. Assessment plans include a listing of measures that drug manufacturers need to address in their scheduled assessment reports, often at 6-, 12-month, and annually. Each assessment plan is tailored to each REMS program which results in variability in number and type of measures assessed per program. The original approval letters represent measures pre-specified at the time of approval.

Adaptation of the RE-AIM Dimensions

Using the established RE-AIM framework (20, 21), the authors Toyserkani (GT), Huynh (LH), and Morrato (EM) created construct definitions applicable to REMS assessments by adapting from those defined by the framework as shown in Table 2.

The adaptation for applying RE-AIM to assessment of REMS was informed by the Scoring Instrument developed for assessing NCI Research-Tested Intervention (RTIPs) programs (19). RTIPs is a searchable database of evidence-based cancer control interventions and program materials and is designed to provide program planners and public health practitioners easy and immediate access to research-tested materials. The adaptation was also informed by the RE-AIM checklist for “RE-AIM Dimensions When Evaluating Health Promotion Programs and Policies” found at RE-AIM.org (24).

The goal was to be as consistent with the constitutive definitions of the RE-AIM dimensions as possible. For example, Adoption was defined as the number and proportion of healthcare settings and providers that agree to initiate program or policy change and how representative they are of the intended audience in terms of the setting and the staff. As REMS programs are multi-level interventions, dimensions were further delineated based on the healthcare setting (system level), healthcare provider (agent level), and patient (individual level).

The dimensions were then mapped to categories outlined in the Assessment Guidance to discern the ease of mapping RE-AIM to the Guidance and determine where opportunities in the assessment process may exist.

Coding

Three blinded reviewers (GT, LH, EM) adjudicated the application of RE-AIM dimensions by coding each REMS assessment measure for three randomly selected REMS assessment plans (average IRR = 75%). They discussed coding discrepancies and refined the dimensions' definitions accordingly. Two blinded reviewers (GT, LH) then categorized each assessment measure(n = 540) for the remaining 15 assessment plans with a third reviewer (EM) serving as an adjudicator.

Analysis

Sensitivity analyses were performed to examine qualitative differences over time, by type of application (NDA vs. BLA), and by type of ETASU required. Descriptive statistics were calculated to determine the proportions of RE-AIM dimensions per REMS assessment program. The median number of assessment measures were then independently analyzed by the variables: year approved, application type, and ETASU to identify any correlations.

Using the alignment between RE-AIM dimensions and Assessment Guidance categories, each assessment measure was then assessed for its inclusion of the categories. This was done by noting how many assessment measures were reflective of each category and then measuring these individual values against the total number of assessment measures for each program. Aggregate summary statistics were reported for the number of measures per category and frequency distribution across all programs.

Results

A total of 18 REMS programs involving nine NDAs and nine BLAs met evaluation eligibility criteria. Table 3 shows the characteristics of the REMS programs meeting criteria at the time of their original REMS approval. Programs by year approved ranged from two in 2016 to five in 2018. The drug products carried a variety of risks intended to be mitigated by the REMS, ranging from cancers such as lymphoma and osteosarcoma, immune system disorders such as autoimmune conditions and cytokine release syndrome, and psychiatric disorders such as suicidal ideation and behavior. The number of assessment measures per program ranged from 10 to 57.

TABLE 3

Table 3. Characteristics of selected active, single product REMS with ETASU programs at time of original approval (2014–2018) included for content analysis of assessment plans using the RE-AIM framework.

Frequency Distribution Analysis of RE-AIM Dimensions

Table 4 shows the distribution of REMS programs (N = 18) and assessments measures (n = 540) across the RE-AIM dimensions. The 18 programs yielded a total of 540 assessment measures; of these, only three measures (0.6%) could not be mapped to a single RE-AIM dimension. These included measures where the intent was unclear or there were multiple intents of the assessment measure that it could have been categorized into more than one dimension.

TABLE 4

Table 4. Distribution of REMS programs and assessment measures across RE-AIM dimensions.

Of 18 REMS programs, the median number of assessment measures assessing Reach per assessment plan was 2 (IQR: 2–4). Prototypical examples of REMS assessment measures categorized as assessing reach included “age and gender of enrolled patients” and “total number of orders shipped to pharmacies.”

Similarly, the median number of assessment measures assessing Effectiveness per REMS assessment plan was 2.5 (IQR: 1–4). Prototypical examples of REMS assessment measures categorized as assessing effectiveness included “adverse event assessments” and “an evaluation of knowledge of patients of the increased risks.”

Regarding Adoption, the median number of assessment measures assessing this dimension per REMS assessment plan was 3.5 (IQR: 2–5). Prototypical examples of REMS assessment measures categorized as assessing adoption included the “number of newly enrolled and active pharmacies stratified by type of pharmacy and geographic location” and “number and location of REMS training programs.”

The median number of assessment measures assessing Implementation per REMS assessment plan was 18 (IQR: 15–24). Prototypical examples of REMS assessment measures categorized as assessing implementation included: “date when the REMS website went live,” “summary report of program problems reported and corrective actions resulting from issues identified,” and “number of prescriptions written by non-certified prescribers and detailed root-cause analysis.”

Finally, the median number of assessment measures assessing Maintenance per REMS assessment plan was 0 (IQR: 0–1). Prototypical examples of REMS assessment measures categorized as assessing maintenance included: “number of discontinued patients” and “number of healthcare settings re-enrollments and the expected number of re-enrollments.”

Figure 2 shows a lack of time trends in RE-AIM dimensions by year of REMS approval. No trends in the number or distribution of RE-AIM dimensions were observed by drug application type or specific ETASU element required.

FIGURE 2

Figure 2. Median number of REMS assessment measures per RE-AIM dimension by year (2014–2018).

Alignment With FDA Assessment Guidance

Consistent with the adaptation of RE-AIM to REMS, the application of RE-AIM dimensions to the Assessment Guidance demonstrated heavy focus on Implementation and Operations. Because assessment measures categorized into the Implementation dimension could be measuring either the healthcare provider's knowledge or processes, these were categorized as either Implementation and Operations (n = 315, 58%) or Safe Use Behaviors and Knowledge (n = 52, 10%) according to the Assessment Guidance. Likewise, measures categorized into the Effectiveness dimension were categorized as Safe Use Behaviors and Knowledge (n = 20, 4%) if they assessed knowledge, and Health Outcomes (n = 30, 5%) if they measured patient understanding. Outreach and Communications (n = 112, 22%) were akin to Reach or Adoption depending on the target audience. Measures of maintenance (n = 8, 1%) were lacking in REMS assessments.

Discussion

To our knowledge, this is the first systematic content analysis examining the feasibility and utility of applying an implementation science framework across a range of REMS programs. The application of social science theories and frameworks to pharmaceutical risk minimization design, implementation and evaluation has been discussed by Smith and Morrato (25). Others have proposed a variety of implementation measures for health services research and pharmaceutical risk minimization evaluation (26–28). Theories and frameworks can provide a social science mechanism of action to understand the relationship between measures and the causal pathway affecting the success of REMS programs in much the same way that a biological mechanism of action guides the clinical development of new medicines. Even risk minimization programs that address only a subset of constructs with a theoretical model can be framed conceptually, so that regulators and the public perceive the larger context and body of literature guiding these programs (29). Ultimately, the use of theories and frameworks helps enable cross-program comparisons and foster generalizable knowledge to advance the science of risk mitigation dissemination and implementation.

Our pragmatic application of RE-AIM to REMS assessment plans were feasible and relatively intuitive to perform. The primary challenge was defining who is the program recipient and who is the program agent, given the complex and multi-faceted nature of REMS programs and their systems-, provider- and patient-level involvement. As Glasgow et al. have defined the product of Reach and Effectiveness to be the individual level, we interpreted this to refer to patients—the recipient of the program or “groups receiving” the intervention (21). Adoption and Implementation then corresponded to the agents of the program, or “staff members” and “program-level” participants who facilitate the delivery of the program to the patients. Glasgow identified the product of these two dimensions to be at the organization level, which we construed to apply to the healthcare providers and healthcare settings. Once we established this distinction of program recipient vs. agent, determining the RE-AIM dimension of each REMS assessment measure was intuitive.

Our findings demonstrate strong congruence between the RE-AIM framework and REMS assessment measures. Consistent with the Assessment Guidance statement that “REMS can be assessed using both process indicators and the intended outcomes,” application of RE-AIM to REMS assessment plans found heavy emphasis on Implementation and Operations measures (10). However, this research also detected lighter emphasis on Health Outcomes measures in REMS assessment plans. Health outcomes in general are difficult to assess especially given rare adverse drug events. For the majority of drugs, FDA relies on routine pharmacovigilance and spontaneous adverse event reporting(passive surveillance) received through post-market periodic safety update reports (PSURs) (30). However, under-reporting is a major drawback and may underestimate the number of adverse drug events (30, 31). In certain REMS programs, FDA does require patient registries, such as pregnancy registry for drugs with risk of birth defects, that collect case data on safety events (active surveillance) (32). Given that health outcomes are generally challenging to assess, assessment of REMS program effectiveness typically relies on process measures such as knowledge attainment and safe use behaviors.

This research also observed very low inclusion of Maintenance measures in REMS assessment plans. Adding measures of Maintenance to REMS assessment plans can strengthen the quality of REMS programs. Maintenance measures represent an area for real-world evidence of REMS integration into the healthcare system and its sustainability. For example, these metrics would measure the durability of knowledge of healthcare providers or cumulative enrollment of healthcare providers in a program over time or evaluate for the evidence of the integration of REMS processes and procedures into state and institutional policies, treatment guidelines, insurance requirements. Our findings are very similar to conclusions reached by previous studies citing that maintenance and representativeness were reported much less often in other health intervention evaluations (33).

The RE-AIM framework offers a number of strengths, including the fact that it considers representativeness and characteristics of the participants to assess heterogeneity of impact. By assessing heterogeneity of impact, RE-AIM permits evaluations of patient access, healthcare system, and patient burden, a potential unintended consequence of risk mitigation requirements that is of public stakeholder interest. RE-AIM also addresses the often-neglected goal of long-term maintenance at both the individual and organization levels. Finally, RE-AIM considers both process and outcome measures, covering the scope of many domains of interest for REMS.

Another important strength is using frameworks like RE-AIM can help REMS assessments be more transparent and better understood across all stakeholders, as it was originally intended. Having a framework for evaluating REMS can facilitate standardization, consistency, and completeness in assessing REMS to enable comparisons across programs (33). By using the commonly recognized constructs and terminology of RE-AIM, data collected by the REMS program can be more meaningful.

Our application of RE-AIM demonstrated some challenges of the framework. The first challenge is one of definition. It has been acknowledged that RE-AIM application has “frequent issues with confusing different dimensions” (33). For example, as aforementioned, defining the agent and recipient of the program is open to interpretation. Others suggest defining Reach at the healthcare provider-level, not the patient-level as we did (34). Effectiveness has been applied to non-patient stakeholders, such as the healthcare provider and pharmacists, at times “requir[ing] multiple creative and innovative combinations of metrics” (34, 35). This contrasts our interpretation of Effectiveness to apply only at the patient-level, which, as aforementioned, was to be as consistent with the constitutive definition of RE-AIM as possible (24).

The second challenge is one of longitudinal assessment. It is not readily apparent how best to apply the framework in a longitudinal and time-dependent manner, although RE-AIM has been proposed for evaluating adaptation over time (33). Pharmaceutical risk management consists of the iterative process of assessing a product's risk-benefit balance, developing and implementing tools, evaluating the tools, and making adjustments to maintain or improve the benefit-risk balance (36). A single REMS program may be implemented for decades as long as the drug product remains on the market. Moreover, the FDAAA requires that assessments be conducted at 18 months, 3 years, and 7 years post-market, at a minimum (13). FDA has required more frequent assessments for REMS with ETASU. Further research is needed to elucidate the pragmatic use of the framework during the REMS life cycle by aligning and differentiating specific RE-AIM measures at different time points of adoption. For example, what are early markers of Effectiveness vs. later markers? What are early markers of Maintenance vs. later markers?

The third challenge is one of utility for decision making. Regulators need to use assessment data to determine whether to sustain, modify or eliminate a REMS program. Should a REMS regulatory determination require a collective gestalt of all RE-AIM dimensions or rely on a single dimension, and if so, how might that best be accomplished in a standardized manner? Our research, similar to previous applications of RE-AIM, found that not all dimensions were assessed equally (18). This observation raises the question of whether all five dimensions are of equal importance, or are there dimensions that are more important, when determining whether a REMS program is meeting its public health drug safety goals.

The limitations of our study include examining only the RE-AIM framework to characterize REMS assessment plans. Future work should evaluate the application of other established frameworks such as PRECEDE-PROCEED, Consolidated Framework for Implementation Research (CFIR), and Practical, Robust Implementation Sustainability Model (PRISM) (33, 37–39). Secondly, this study looked at REMS assessment plans from 2014 to 2018 and does not consider the potential impact or evolution of REMS assessment plans since the publication of the Assessment Guidance issued in 2019. Furthermore, assessment measures from Shared Systems REMS (multiple products of the same class or molecular moiety under two or more sponsors) can offer additional insights into the strengths and opportunities for REMS assessments.

Of note, our study examined the type and quantity of REMS assessment measures from the original approval; however, it did not assess the rigor of proposed study designs nor the quality of their reports. Reporting standards for risk minimization communication and program evaluation have been described by members of the International Society for Pharmacoepidemiology (40). A systematic review of the published literature on pharmaceutical risk minimization evaluation found limited use of conceptual frameworks guiding process and outcome measurement selection and program design and implementation (41).

FDA considers public comments and stakeholder feedback as it finalizes guidance to industry. Therefore, learning from the current analysis is one source of input that FDA is considering as it works on the final version of the REMS Assessment Guidance affecting all future REMS programs. The guidance aims to ultimately improve how REMS assessment plans are developed, specifically how the REMS program goals, objectives and REMS design may impact the selection of metrics and data sources, which will be used to assess whether the program is meeting its risk mitigation goals (16).

In addition, findings are also relevant to FDA's efforts on structured benefit-risk assessment process and commitments established in the sixth authorization of the PDUFA VI in 2017. FDA has made several commitments in PDUFA VI for continued implementation of structured benefit-risk assessment, including publishing a draft guidance on benefit-risk assessment. “Risk and Risk Management” is one explicit dimension in the benefit-risk assessment framework (42). How information, including evidence and uncertainties, can be effectively communicated to the public is one area of interest. To meet requirements established in the 21st Century Cures Act, the benefit-risk assessment guidance will also discuss how relevant patient experience data may be used to inform benefit-risk assessments (43). RE-AIM provides a natural structured analytic approach for synthesizing risk management effectiveness evidence and integrating patient data into the assessment.

Conclusion

Dissemination and implementation science frameworks can provide a systematic approach for REMS program assessments. They provide a structured and evidence-based approach to guide what should be evaluated and to what extent, and to identify which aspects of the programs will be considered when judging REMS program performance, including a priori expectations for program success. Frameworks like RE-AIM, can be readily applied to REMS assessments to strengthen their evaluation and have the potential to advance science, quality of practice, and population health through all participants affected by REMS.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Funding

GT was employed by the U.S. Food and Drug Administration. LH was supported by an appointment to the Research Participation Program at the U.S. Food and Drug Administration administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy. EM was supported by an Intergovernmental Personnel Act appointment to the U.S. Food and Drug Administration, Office of Surveillance and Epidemiology administered through the University of Colorado.

Disclaimer

This article reflects the views of the authors and should not be construed to represent FDA's views or policies.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Results were presented in part at the 35thInternational Conference of Pharmacoepidemiology and Therapeutic Risk Management, August 28, 2019. Philadelphia, USA (44).

References

1. About FDA. Available online at: https://www.fda.gov/about-fda (accessed August 30, 2019).

2. IOM. To Err Is Human. Washington, DC: The National Academies Press (2000).

3. PDUFA Reauthorization Performance Goals and Procedures Fiscal Years 2018 Through 2022. Available online at: https://www.fda.gov/media/99140/download (accessed August 30, 2019).

4. FDA. Classifying Significant Postmarketing Drug Safety Issues. (2012). Available online at: https://www.fda.gov/media/83087/download

5. National Action Plan for Adverse Drug Event Promotion. Washington, DC (2014). Available online at: https://health.gov/hcq/pdfs/ADE-Action-Plan-508c.pdf (accessed August 30, 2019).

6. Food and Drug Administration Amendments Act of 2007. HR, 110th Congress Session. House Committee on Energy and Commerce (2007).

7. FDA. FDA's Application of Statutory Factors in Determining When a REMS Is Necessary. Drug Safety (2019). Available online at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/fdas-application-statutory-factors-determining-when-rems-necessary

8. FDA. Approved Risk Evaluation and Mitigation Strategies (REMS). Available online at: https://www.accessdata.fda.gov/scripts/cder/rems/index.cfm (accessed September 3, 2019).

9. FDA. Medication Guides — Distribution Requirements and Inclusion in Risk Evaluation and Mitigation Strategies (REMS). Drug Safety (2011). Available online at: https://www.fda.gov/media/79776/download

10. FDA. REMS Assessment: Planning and Reporting. (2019). Available online at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/rems-assessment-planning-and-reporting

11. CRS. The SUPPORT for Patients and Communities Act (P.L. 115-271): Food and Drug Administration and Controlled Substance Provisions (2018).

12. FDA. Fixed-Quantity Unit-of-Use Blister Packaging for Certain Immediate Release Opioid Analgesics for Treatment of Acute Pain; Establishment of a Public Docket; Request for Comments. (2019). Available online at: https://www.govinfo.gov/content/pkg/FR-2019-05-31/pdf/2019-11283.pdf

13. U.S. Congress. United States Code: Federal Food, Drug, and Cosmetic Act, 21 U.S.C. ßß301–392 (Suppl. 5. 1934). Periodical. Available online at: https://www.loc.gov/item/uscode1934-006021009/

14. FDA. Opioid Analgesic Risk Evaluation and Mitigation Strategy (REMS). Information by Drug Class. Available online at: https://www.fda.gov/drugs/information-drug-class/opioid-analgesic-risk-evaluation-and-mitigation-strategy-rems (accessed September 27, 2018).

15. FDA. Zyprexa Relprevv (olanzapine). Approved Risk Evaluation and Mitigation Strategies (REMS). Available online at: https://www.accessdata.fda.gov/scripts/cder/rems/index.cfm?event=IndvRemsDetails.page&REMS=74 (accessed April 11, 2019).

16. FDA. Risk Evaluation and Mitigation Strategies Assessment: Planning and Reporting; Draft Guidance for Industry; Availability. (2019). Available online at: https://www.govinfo.gov/content/pkg/FR-2019-02-01/pdf/2019-00676.pdf

17. Frame JN, Jacobson JO, Vogel WH, Griffith N, Wariabharaj D, Garg R. Assessment of risk evaluation and mitigation strategies in oncology: summary of the oncology risk evaluation and mitigation strategies workshop. J Oncol Pract. (2012) 9:24–39. doi: 10.1200/JOP.2012.000620

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am. J. Public Health. (2013) 103:38–46. doi: 10.2105/AJPH.2013.301299

PubMed Abstract | CrossRef Full Text | Google Scholar

19. NCI. RE-AIM Scoring Intsrument. Available online at: https://rtips.cancer.gov/rtips/reAim.do (accessed September 04, 2013).

20. Workgroup R-A. What is RE-AIM? The Basics. Available online at: http://www.re-aim.org/about/frequently-asked-questions/

21. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am. J. Public Health. (1999) 89:1322–7.

PubMed Abstract | Google Scholar

22. Stoutenberg M, Galaviz KI, Lobelo F, Joy E, Heath GW, Hutber A, et al. A pragmatic application of the RE-AIM framework for evaluating the implementation of physical activity as a standard of care in health systems. Prev Chronic Dis. (2018) 15:E54. doi: 10.5888/pcd15.170344

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Jilcott S, Ammerman A, Sommers J, Glasgow RE. Applying the RE-AIM framework to assess the public health impact of policy change. Ann Behav Med. (2007) 34:105–14. doi: 10.1007/bf02872666

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Workgroup R. Measures and Checklists. (2019). Available online at: http://www.re-aim.org/resources-and-tools/measures-and-checklists/

25. Smith MY, Morrato E. Advancing the field of pharmaceutical risk minimization through application of implementation science best practices. Drug Safety. (2014) 37:569–80. doi: 10.1007/s40264-014-0197-0

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Rabin BA, Purcell P, Naveed S, Moser RP, Henton MD, Proctor EK, et al. Advancing the application, quality and harmonization of implementation science measures. Implement Sci. (2012) 7:119. doi: 10.1186/1748-5908-7-119

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Prieto L, Spooner A, Hidalgo-Simon A, Rubino A, Kurz X, Arlett P. Evaluation of the effectiveness of risk minimization measures. Pharmacoepidemiol Drug Saf. (2012) 21:896–9. doi: 10.1002/pds.3305

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. (2011) 38:65–76. doi: 10.1007/s10488-010-0319-7

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Proctor EK, Powell BJ, Baumann AA, Hamilton AA, Santens RL. Writing implementation research grant proposals: ten key ingredients. Implement Sci. (2012) 7:96. doi: 10.1186/1748-5908-7-96

PubMed Abstract | CrossRef Full Text | Google Scholar

30. FDA. Providing Postmarketing Periodic Safety Reports in the ICH E2C(R2) Format (Periodic Benefit-Risk Evaluation Report). Drug Safety (2016). Available online at: https://www.fda.gov/media/85520/download

31. Alvarez-Requejo A, Carvajal A, Bégaud B, Moride Y, Vega T, Martin Arias LH. Under-reporting of adverse drug reactions Eur J Clin Pharmacol. (1998) 54:483–8.

32. FDA. Risk Evaluation and Mitigation Strategies (REMS): The iPLEDGE Program. (2018). Available online at: https://www.accessdata.fda.gov/drugsatfda_docs/rems/Isotretinoin_2018_04_23_REMS_Document.pdf

Google Scholar

33. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Chan-Liston M, Bourke J, Sheehan P, Niemcryk S, Peng M, Bwire R, et al. 17th ISoP Annual Meeting “Pharmacovigilance in the 21st century” Liverpool, UK 15–18 October, 2017 Drug Safety. (2017) 40:937–1045. doi: 10.1007/s40264-017-0580-8

CrossRef Full Text

35. CDER. Standardizing and evaluating risk evaluation and mitigation strategies (REMS). In: FDA Public Meeting. Silver Spring, MD (2013).

36. Radawski C, Morrato E, Hornbuckle K, Bahri P, Smith M, Juhaeri J, et al. Benefit–Risk Assessment, Communication, and Evaluation (BRACE) throughout the life cycle of therapeutic products: overall perspective and role of the pharmacoepidemiologist. J Pharmacoepidemiol Drug Safety. (2015) 24:1233–40. doi: 10.1002/pds.3859

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4:50. doi: 10.1186/1748-5908-4-50

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Glanz K, Rimer BK, Viswanath K, editors. Using the PRECEDE-PROCEED model to apply to health behavior theories. In: Health Behav Health Education: Theory, Research and Practice, 4 ed. San Francisco, CA: Jossey-Bass (2008). p. 407–33.

Google Scholar

39. Birken SA, Powell BJ, Presseau J, Kirk MA, Lorencatto F, Gould NJ, et al. Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): a systematic review. Implement Sci. (2017) 12:2. doi: 10.1186/s13012-016-0534-z

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Smith MY, Russell A, Bahri P, Mol PGM, Frise S, Freeman E, et al. The RIMES statement: a checklist to assess the quality of studies evaluating risk minimization programs for medicinal products. Drug Saf. (2018) 41:389–401. doi: 10.1007/s40264-017-0619-x

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Russell AM, Mullen RJ, Morrato EH, Smith MY. Quality of reporting on the evaluation of risk minimization programs: a systematic review. Drug Saf. (2020). doi: 10.1007/s40264-020-00905-8. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text

42. FDA. Benefit-Risk Assessment in Drug Regulatory Decision-Making. (2018). Available online at: https://www.fda.gov/files/about%20fda/published/Benefit-Risk-Assessment-in-Drug-Regulatory-Decision-Making.pdf

43. Congress. 21st Century Cures Act. Available online at: https://www.congress.gov/bill/114th-congress/house-bill/34

44. Toyserkani GA, Huynh L, Morrato EH. Characteristics of risk evaluation and mitigation strategies (REMS) assessment plans: a review of REMS with ETASU (2014-2018) using RE-AIM. 35th International Conference on Pharmacoepidemiology & Therapeutic Risk Management; August 20, 2019. Philadelphia, PA: Pennsylvania Convention Center (2019). p. 466.