An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time

RE-AIM is a widely adopted, robust implementation science (IS) framework used to inform intervention and implementation design, planning, and evaluation, as well as to address short-term maintenance. In recent years, there has been growing focus on the longer-term sustainability of evidence-based programs, policies and practices (EBIs). In particular, investigators have conceptualized sustainability as the continued health impact and delivery of EBIs over a longer period of time (e.g., years after initial implementation) and incorporated the complex and evolving nature of context. We propose a reconsideration of RE-AIM to integrate recent conceptualizations of sustainability with a focus on addressing dynamic context and promoting health equity. In this Perspective, we present an extension of the RE-AIM framework to guide planning, measurement/evaluation, and adaptations focused on enhancing sustainability. We recommend consideration of: (1) extension of “maintenance” within RE-AIM to include recent conceptualizations of dynamic, longer-term intervention sustainability and “evolvability” across the life cycle of EBIs, including adaptation and potential de-implementation in light of changing and evolving evidence, contexts, and population needs; (2) iterative application of RE-AIM assessments to guide adaptations and enhance long-term sustainability; (3) explicit consideration of equity and cost as fundamental, driving forces that need to be addressed across RE-AIM dimensions to enhance sustainability; and (4) use or integration of RE-AIM with other existing frameworks that address key contextual factors and examine multi-level determinants of sustainability. Finally, we provide testable hypotheses and detailed research questions to inform future research in these areas.

Introduction

RE-AIM is a robust framework that has been widely applied over the past 20 years across a range of public health, clinical, community, and behavioral settings (1–4). RE-AIM was created to help address the well-documented research-to-practice gap that hinders the reduction of health inequities and widespread population health impact. It is one of the most commonly applied frameworks in public health, health behavior, and implementation science (IS) (2–5). RE-AIM can facilitate transparent reporting (1) and enhance planning for successful dissemination and implementation of evidence-based interventions, programs, practices, and policies (“EBIs”). Recent years have seen expansion of RE-AIM to address contextual factors (e.g., RE-AIM/PRISM); (4, 6) and integrate qualitative methods (7, 8).

As a framework, RE-AIM has both individual-level and staff/setting-level dimensions, including Reach and Effectiveness (individual-level), Adoption and Implementation (staff and setting levels), and Maintenance (both individual and staff/setting levels). Recognizing sustained delivery and impact of EBIs as central challenges across settings, RE-AIM has historically been one of the few IS frameworks that explicitly built in measurement and consideration of “maintenance.” “Maintenance” in RE-AIM has been operationalized at the individual level (e.g., long-term effectiveness or impact of EBI) and the setting level (e.g., sustainability of EBI program components after original implementation). The “maintenance” dimension of RE-AIM has typically been assessed at relatively short-term intervals (e.g., 6 months after EBI delivered or initially implemented) and its evaluation has focused on the extent to which a program/policy becomes institutionalized (e.g., made part of routine organizational practices and policies) (4).

Within IS, there is growing recognition of the importance of understanding and addressing longer-term sustainability of EBIs (9–12). Achieving sustained impact and delivery of EBIs over time has been identified as one of the most important yet understudied challenges across settings, populations, and health issues (9, 10, 13, 14). There is growing consensus on conceptualizations and definitions of sustainability; e.g., Moore et al. (15) described sustainability as “after a defined period of time, the program, clinical intervention, and/or implementation strategies continue to be delivered and/or individual behavior change (i.e., clinician, patient) is maintained; the program and individual behavior change may evolve or adapt while continuing to produce benefits for individuals/systems.” Of note, we recognize that the terms sustainment and sustainability are both used in reference to the outcome of an intervention being delivered over time, as well as the characteristics of the intervention that make it more likely to be delivered over time. For this paper, we used the term “sustainability” to refer to both the desired outcome and the characteristics or processes by which it is more likely maintained.

There has been an important shift away from “static” conceptualizations of sustainability, with awareness that this may impede adoption of more effective practices as the environment changes or new evidence emerges. Investigators also increasingly recognize the need for a dynamic conceptualization of sustainability, in light of complex “real-world” contexts in which EBIs are delivered that require responsiveness, capacity building, and adaptation of EBIs (10, 11, 16). This is consistent with the Dynamic Sustainability Framework (DSF) (17), which focuses on continued learning and evaluation, problem-solving, improvement and ongoing adaptation of EBIs to enhance fit with contexts and populations. Just as a balance between fidelity and adaptation is needed to achieve “fit” in the context of pre-implementation and implementation efforts (18, 19), there is a similar balance between sustainability of original EBIs and ongoing “evolvability” to achieve ongoing fit and sustained population health impact within broader communities or health systems. Evolvability (20) relates to the adaptation of EBIs and implementation strategies in response to changing contexts and resources over time, as well as emerging needs and evidence across the life cycle of an EBI. This includes both the systematic, planned adaptation of EBIs and strategies, as well as ongoing refinement of EBIs and strategies organically within specific community or clinical settings. Over an EBI's life cycle, this evolution within a changing system or organization may ultimately involve “de-implementation,” or the removal or replacement of EBIs that no longer fit or are ineffective (21, 22).

As explicated below, equity and costs are foundational driving forces across RE-AIM dimensions that shape sustained impact, and warrant the need for initial and ongoing adaptation. EBIs can only succeed at the population health level if they are affordable across most settings and are delivered routinely and equitably over time across diverse settings and populations. As we consider the life cycle of an intervention (18), it may be less useful to think about “sustainability” of the original EBIs as an “end goal” (17), and instead consider “evolvability” across the dynamic life cycle of the EBI within a broader context or system, with the goal of sustainable and equitable health impact.

Important gaps persist in existing frameworks' ability to provide guidance in concretely conceptualizing, measuring/operationalizing, and planning for longer-term sustainability within a dynamic context. For example, RE-AIM does not capture such dynamic conceptualizations of sustainability, and has often been applied as a “one-time” evaluation and planning tool. Given the numerous conceptual frameworks and models in IS (2, 23, 24), we did not seek to create a new framework. Instead, we propose an expansion of RE-AIM to enhance sustainability by focusing on key issues across RE-AIM dimensions, with the goal of increasing health impact and health equity over time.

The purposes of this article are to: (1) discuss the extension of RE-AIM to address dynamic conceptualization of sustainability over time, including iterative application of RE-AIM to guide adaptation and evolvability of EBIs and implementation strategies; (2) provide concrete guidance on issues pertinent to understanding, measuring, and planning for sustainability in changing context, including explicit consideration of costs and equity; and (3) propose testable hypotheses and detailed research questions to guide future research that applies RE-AIM for sustainability.

Applying Re-Aim to Enhance Sustainability

The following sections discuss and provide recommendations to guide planning, adaptation, and measurement when applying RE-AIM to facilitate sustainability, reflecting dynamic sustainability with a focus on context and equity. Each section concludes with example hypotheses to guide research. Five key issues are discussed below, and summarized in Table 1 and Figure 1.

TABLE 1

Table 1. Iterative application and operationalization of RE-AIM for Sustainability, with a focus on health equity and dynamic context over time.

FIGURE 1

Figure 1. An extension of RE-AIM to enhance sustainability: Cross-cutting issues and iterative application of RE-AIM for sustainability, to guide adaptations and evolvability of EBIs/implementation strategies, address dynamic context, and promote equity across the life cycle of an EBI.

1. Extending and Reframing “Maintenance” Within RE-AIM to Include Recent Conceptualizations of Sustainability as an Outcome

Given growing consensus of sustainability as dynamic in nature, it is important that indicators of sustainability reflect this longer-term conceptualization. While 6 months, as originally proposed in RE-AIM (1), is useful in providing an indicator of early maintenance, a more comprehensive approach to also capturing sustainability over time includes measurement at least 1 year post initial implementation and over time (e.g., quarterly to annually) (9, 10).

Consistent with recent conceptualizations (10–12), we recommend that operational indicators of maintenance/sustainability include (see Table 1 for details): (1) extent to which the core components/functions of EBIs and implementation strategies continue to be delivered over time with fidelity (e.g., continuation of active ingredients and essential functions/related activities) (25, 26), and the “evolvability” of the EBI and implementation strategies (27) needed to support continued EBI delivery over time, including adaptations (planned and organic) and why they occur; (2) extent to which the EBI has continued impact on health behaviors/outcomes, when feasible, including patterns in health inequities over time (e.g., who continues to experience health benefits and who does not); and (3) extent to which community and organizational capacity and infrastructure to deliver the EBI are maintained, including partnerships, networks, and coalitions. It is critical to actively engage with stakeholders (e.g., community members, implementers, organizational leaders) to prioritize which maintenance/sustainability outcomes will be measured and when (e.g., which are meaningful and pragmatic to assess and how often). We also recognize the challenges of including ongoing measurement of the effectiveness of EBIs; while in some cases, existing resources may provide data to monitor frequent and continued impact on health behaviors/outcomes, we realize this may not feasible across all settings.

Example Hypothesis: Informed by a broadened, longer-term conceptualization of sustainability, the dose and nature of implementation strategies needed to initially implement an EBI will differ from the strategies needed to sustain an EBI over time (e.g. implementation strategies focused on sustainability may relate to providing proactive planning and ongoing evaluation/monitoring to manage likely changes in the implementation setting, including turnover, EHR upgrades, treatment guideline updates, changes in patient population).

2. To Facilitate Sustainability, Planned Adaptations, and Evolutions Must Be Made Across the Life Cycle of EBIs to Respond to Changing Context

In many cases it is neither feasible nor optimal to continue to deliver the same EBI “protocol” with high fidelity, as context changes over time and across settings. There is often a need in early program stages to make planned “fidelity-consistent” adaptations that reflect diverse settings, cultures, and populations in which they are delivered (16, 18, 28). Failing to make planned cultural or contextual adaptations may have adverse impact on effectiveness, and ultimately, perpetuate health inequities (28, 29). EBIs and implementation strategies that are not aligned with and do not reflect changing community needs, culture, and context are unlikely to be sustained or have sustained impact over time.

It is also likely that there will be evolving evidence (e.g., guidelines change, new populations are exposed to EBI with varied results), setting changes (e.g., staff turnover/attrition; resources change), and shifting population health needs over time that require ongoing adaptations or refinements over time. We recommend proactively planning for adaptations, and documenting why adaptations are needed, and the extent to which EBIs and implementation strategies evolve over the life cycle of a program (16). Iterative application and measurement of RE-AIM dimensions over time enables documentation of effective adaptations to retain fit with ever evolving context. De-implementation (e.g., the removal or replacement of low-value, harmful, costly or non-evidence-based care/EBIs (21, 22), including the need to make a program and its delivery less expensive), may also be necessary and should be tracked to inform changes in implementation.

Example Hypothesis: Settings that maintain core functions of EBIs but include proactive, planned, iterative adaptations to intervention components and implementation strategies in response to changing context and needs will be sustained longer than those that do not, and will have greater impact on reducing health inequities.

3. Assessment and Feedback on RE-AIM Indicators as an Iterative Method to Guide Adaptations

Assessment of RE-AIM dimensions can help guide settings on how to proactively monitor or adapt and may identify early indicators of sustainability challenges, including the need to “change course” to promote the sustainability of EBIs over time. Results on RE-AIM dimensions should not be assumed to be static. Thus, as explicated in Table 1, RE-AIM indicators (e.g., Reach, Effectiveness) should be measured repeatedly and iteratively when possible to provide insight into how to achieve sustained health impacts (4, 30), monitor progress, and shed light on where and when both equity and sustainability issues arise (e.g., over time, which populations and settings is the intervention reaching, and why or why not?).

These findings may impact the nature and timing of actionable solutions across RE-AIM dimensions and program life cycle —e.g., adapt the recruitment or implementation strategies. RE-AIM qualitative probes (8) can also be used to (a) help ensure that the perspectives of key stakeholders and community members are being assessed regularly; and that (b) stakeholders are being actively engaged in planning for sustainability in ways consistent with their values (e.g., “What would it take for you/your organization/your community to sustain the EBI over the long term?”) (12). Aligned with existing taxonomies of implementation strategies (e.g., evaluation/iterative strategies) (27, 31), RE-AIM can be used as a tool to complement existing quality improvement (QI) and performance management resources (e.g., PDSA cycles) (32, 33). As such, iterative application of RE-AIM can provide guidance and a conceptually-based, standardized evaluation approach to understand what is working or not; this information can be used to inform QI activities (e.g., who participates and why; where in the system is implementation of highest/lowest quality), with implications for long-term sustainability and impact.

Example Hypothesis: Programs that iteratively assess and address RE-AIM dimensions over time to guide their sustainability planning and adaptations will have stronger sustainability outcomes (e.g. higher levels of continued delivery of EBI; higher levels of sustained behavior change across population groups) than those that do not.

4. Other Sustainability Frameworks Can Be Integrated With RE-AIM to Understand Key Sustainability Determinants

While several frameworks provide consideration of multi-level contextual factors that influence sustainability, many have been most explicitly applied in the context of implementation (34) e.g., PRISM (6, 35); and Consolidated Framework for Implementation Research [CFIR; (36)]. There may be value in also considering frameworks that have focused specifically on sustainability, including the Program Sustainability Assessment Tool [PSAT; (37)] and the Integrated Sustainability Framework [ISF; (11)], which provide a strong foundation for understanding multi-level contextual determinants of sustainability, but less guidance in measuring sustainability outcomes, or thinking explicitly about dynamic sustainability. These multi-level determinant frameworks may call attention to constructs that are particularly important to sustainability (e.g., sustainability planning, funding stability, staff retention over time), and can be integrated with RE-AIM to inform questions, measurements and actions related to contextual determinants of sustainability. For example, the ISF could be used to understand and assess multi-level aspects that may influence sustainability- e.g., “How have program champions played a role in sustaining the EBI?” Data from such qualitative assessments would preferably be integrated with quantitative measures of sustainability determinants (e.g., informed by the ISF or PSAT). It is important to recognize that sustainability determinants themselves are likely not static, and may change over time.

Example Hypotheses: 1) Programs that explicitly address multi-level contextual determinants of sustainability will produce higher levels of sustainability and equity than those that do not; 2) Programs that address changing multi-level context and determinants of sustainability will be sustained longer than those addressing only one level.

5. Focus on Costs and Equity as Key Drivers of Sustainability Can Inform and Guide Dynamic Sustainability

Promoting health equity¹ (39, 40) is a central part of our conceptualization and measurement of sustainability, and RE-AIM indicators should be tracked over time to identify and address inequities when they arise (further explicated in Table 1). All RE-AIM dimensions include representativeness (heterogeneity, generalization), which should be assessed across different types of patient/population subgroups of focus (e.g., by race/ethnicity, age, disability, insurance status, literacy level, social determinants of health), and settings (e.g., urban/rural, lower vs. higher resource settings). Consistent with notions of “equitable implementation” (40), it is critical to document and address inequities as they emerge across all RE-AIM dimensions. Not doing so risks maintaining or even exacerbating health inequities, and ultimately inequitable use of EBIs over time.

Issues of cost and resources required are strongly tied to health equity. For example, if an EBI is not feasible for delivery in certain settings (e.g., community health centers) due to constrained resources or insufficient staff, inequities may result. This is because these settings often reach populations that experience disproportionate social stressors and greater structural barriers to care. At the individual level, if participation requires considerable costs or burden such as travel or time off work, unintentional health inequities may result. To prevent such consequences, initial cost estimates and resource requirements should be discussed with stakeholders at the planning stage, and costs can be periodically assessed, discussed and necessary adaptations made over time (41, 42).

We consider “costs” very broadly, including understanding, planning for, and tracking economic costs, time, resources, burdens, and unintended political and social consequences (e.g., social stigma) of an EBI, especially from the perspectives of different stakeholders (e.g., implementers, administrators, community members, and patients). Recent IS research (42–44) provides suggestions for cost assessment to understand the impact on sustainability. We also encourage consideration of economic factors more broadly, including the “value” and return on investment of sustaining the EBI, and the priorities of, and value to, different stakeholders (42), including community partners.

Example Hypotheses: 1) Programs that explicitly and repeatedly assess health equity and equitable implementation, and make iterative adjustments guided by RE-AIM will produce higher levels of sustainability than those only considering equity at the planning stage. 2) Programs that consider and monitor costs (and RE-AIM outcomes), ‘return on investment' over time, and discuss and act on these assessments in partnership with stakeholders will produce stronger sustainable outcomes than those that do not.

Summary

The discussion above illustrates key issues involved in extending RE-AIM to enhance sustainability. In Table 1, we outline key indicators, guiding questions, and equity and sustainability considerations in applying this extension and iterative application of RE-AIM. Consistent with complex adaptive systems (45, 46), it is more complex than the discussion makes it appear, as the various factors above and the RE-AIM dimensions are interrelated. Thus, we need to consider interactions among the issues above and across RE-AIM dimensions over time. Figure 1 highlights this complexity and considerations for cross-cutting, intersecting issues and indicators that shift (like gears) over time in different combinations to guide RE-AIM for sustainability in dynamic context across the life cycle of an EBI. This summary figure illustrates the impact of EBIs and implementation strategies on the RE-AIM dimensions, and how factors such as health equity and costs influence the likelihood of sustainability across the phases of a program.

Discussion

This paper encourages iterative application of RE-AIM with early guidance on understanding, evaluating, and planning for sustainability, with a focus on changing context and health equity. While RE-AIM has previously been applied to promote health equity, this paper reinforces the importance of this focus within the context of sustainability. It advances the IS field beyond existing models and prior RE-AIM publications by providing: (1) consideration of planning for sustainability throughout the life cycle of an EBI and across multiple RE-AIM dimensions; (2) concrete guidance for operationalizing the dynamic and complex nature of sustainability, including the “evolvability” of an EBI and where adaptations and de-implementation may fit within this conceptualization; (3) attention to iterative measurement of RE-AIM indicators to inform and enhance sustainability, and (4) explicit consideration of health equity and costs/value as critical components of sustainability. In summary:

1. Measuring “maintenance” as a RE-AIM dimension is important, but needs to be expanded to address longer-term conceptualizations of sustainability. The conceptualization of dynamic sustainability includes consideration of “evolvability” across the life cycle of an EBI, including continued delivery of the original EBI functions and implementation strategies, adaptations, and potential de-implementation across the EBI life cycle to produce sustained and equitable health outcomes.

2. Multi-level context changes and so must EBIs and implementation strategies to meet emerging needs, resources and challenges over time. Iterative (or at least periodic) use of actionable RE-AIM assessments can guide adaptations to enhance sustainability and respond to changing context.

3. Equity (both equitable implementation across RE-AIM dimensions and health equity) and costs/value are important and understudied cross-cutting issues across all RE-AIM dimensions that impact sustainability. Researchers should assess and address these factors in planning for and facilitating long-term sustainability.

This article has both strengths and limitations. Strengths include its focus on costs and value, from the perspective of multiple stakeholders, and health equity and representativeness across all RE-AIM dimensions as key drivers of sustainability. Additionally, instead of proposing another IS model, we provide an extension of, and guidance from, a widely adopted IS framework. This paper and our recommendations address sustainability processes and planning, as well as sustainability outcomes. Finally, we make recommendations and testable hypotheses that should lead to incremental validation, revision or rejection as we refine this extension of RE-AIM. Limitations include that this proposed expansion of RE-AIM needs further empirical support. We call for future application across diverse health issues and settings, and mixed-methods research to investigate and refine this extension of RE-AIM for sustainability. There is still much to learn about sustainability, and we believe this application will provide a useful guide and addition to the IS literature.

Author Contributions

RS and RG initially conceptualized the paper. RS took the lead in writing an initial draft. All authors contributed to reviewing, revising, and rewriting sections for this Perspective piece.

Funding

RS was funded by a Research Scholar Grant from American Cancer Society (RSG 17-156-01-CPPB). RG's contributions were supported by NHLBI grant 5K12 HL137862 and NCI grant 1P50CA244688-01.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We'd like to thank Laura Brotzman for design of the figure and assistance reviewing and preparing this manuscript, and Matthew Lee and Bryan Ford for their assistance in reviewing and preparing this manuscript.

Footnotes

1. ^“Health equity means that everyone has a fair and just opportunity to be as healthy as possible. This requires removing obstacles to health such as poverty, discrimination, and their consequences, including powerlessness and lack of access to good jobs with fair pay, quality education and housing, safe environments, and health care…For the purposes of measurement, health equity means reducing and ultimately eliminating disparities in health and its determinants that adversely effect excluded or marginalized groups” (38).

References

1. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/AJPH.89.9.1322

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. (2012) 43:337–50. doi: 10.1016/j.amepre.2012.05.024

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Harden SM, Smith ML, Ory MG, Smith-Ray RL, Estabrooks PA, Glasgow RE. RE-AIM in clinical, community, and corporate settings: perspectives, strategies, and recommendations to enhance public health impact. Front Public Health. (2018) 6:71. doi: 10.3389/fpubh.2018.00071

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Glasgow RE, Harden SM, Gaglio B, Rabin BA, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a twenty-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

CrossRef Full Text | Google Scholar

5. Vinson CA, Stamatakis K, Kerner J. Dissemination and implementation research in community and public health settings. In: Brownson Rc CG, editor. Dissemination and Implementation Research in Health: Translating Research to Practice. Proctor Ea Oxford University Press (2018). p. 355–70. doi: 10.1093/oso/9780190683214.003.0021

CrossRef Full Text | Google Scholar

6. Mccreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. (2019) 9:1002–11. doi: 10.1093/tbm/ibz085

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. (2018) 15:E02. doi: 10.5888/pcd15.170271

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Holtrop JS, Rabin BA, Glasgow RE. Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Services Res. (2018) 18:177. doi: 10.1186/s12913-018-2938-8

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Scheirer M, Dearing J. An agenda for research on the sustainability of public health programs. Am J Public Health. (2011) 101:e300193. doi: 10.2105/AJPH.2011.300193

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. (2012) 7:17. doi: 10.1186/1748-5908-7-17

CrossRef Full Text | Google Scholar

11. Shelton RC, Cooper BR, Stirman SW. The sustainability of evidence-based interventions and practices in public health and health care. Ann Rev Public Health. (2018) 39:55–76. doi: 10.1146/annurev-publhealth-040617-014731

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Shelton RC, Lee M. Prioritizing sustainability research: innovations and future directions in implementation science. Am J Public Health. (2019) 109:S132–4. doi: 10.2105/AJPH.2018.304913

CrossRef Full Text | Google Scholar

13. Proctor EK, Luke DA, Calhoun A, Mcmillen C, Brownson RC, Mccrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. (2015) 10:88. doi: 10.1186/s13012-015-0274-5

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Johnson AM, Moore JE, Chambers DA, Rup J, Dinyarian C, Straus SE. How do researchers conceptualize and plan for the sustainability of their NIH R01 implementation projects? Implement Sci. (2019) 14:50. doi: 10.1186/s13012-019-0895-1

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Moore JE, Mascarenhas A, Bain J, Straus SE. Developing a comprehensive definition of sustainability. Implementation Sci. (2017) 12:110. doi: 10.1186/s13012-017-0637-1

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Stirman SW, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. (2019) 14:58. doi: 10.1186/s13012-019-0898-y

CrossRef Full Text | Google Scholar

17. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. (2013) 8:117. doi: 10.1186/1748-5908-8-117

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Bopp M, Saunders RP, Lattimore D. The tug-of-war: fidelity versus adaptation throughout the health promotion program life cycle. J Primary Prev. (2013) 34:193–207. doi: 10.1007/s10935-013-0299-y

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Allen JD, Shelton RC, Emmons KM, Linnan L. Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. 2 ed. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. New York, NY: Oxford University Press (2018). p. 267–84. doi: 10.1093/oso/9780190683214.003.0016

CrossRef Full Text | Google Scholar

20. Stange KC. Refocusing knowledge generation, application, and education: raising our gaze to promote health across boundaries. Am J Prev Med. (2011) 41:S164–9. doi: 10.1016/j.amepre.2011.06.022

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Niven DJ, Mrklas KJ, Holodinsky JK, Straus SE, Hemmelgarn BR, Jeffs LP, et al. Towards understanding the de-adoption of low-value clinical practices: a scoping review. BMC Med. (2015) 13:255. doi: 10.1186/s12916-015-0488-z

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Norton WE, Chambers DA, Kramer BS. Conceptualizing de-implementation in cancer care delivery. J Clin Oncol. (2019) 37:93–6. doi: 10.1200/JCO.18.00589

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Strifler L, Cardoso R, Mcgowan J, Cogo E, Nincic V, Khan PA, et al. Scoping review identifies significant number of knowledge translation theories, models, and frameworks with limited use. J Clin Epidemiol. (2018) 100:92–102. doi: 10.1016/j.jclinepi.2018.04.008

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatr Res. (2019) 283:11246. doi: 10.1016/j.psychres.2019.06.036

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Blase K, Fixsen D. Core Intervention Components: Identifying and Operationalizing What Makes Programs Work. ASPE Research Brief. Washington, DC: US Department of Health and Human Services (2013).

Google Scholar

26. Jolles MP, Lengnick-Hall R, Mittman BS. Core functions and forms of complex health interventions: a patient-centered medical home illustration. J General Internal Med. (2019) 34:1032–8. doi: 10.1007/s11606-018-4818-7

CrossRef Full Text | Google Scholar

27. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. (2015) 10:21. doi: 10.1186/s13012-015-0209-1

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Baumann A, Cabassa LJ, Stirman SW. Adaptation in dissemination and implementation science. Dissemin Implement Res Health. (2017) 2:286–300. doi: 10.1093/oso/9780190683214.003.0017

CrossRef Full Text | Google Scholar

29. Castro FG, Barrera M, Martinez CR. The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit. Prev Sci. (2004) 5:41–5. doi: 10.1023/B:PREV.0000013980.12412.cd

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Glasgow RE, Battaglia C, McCreight M, Ayele RA, Rabin BA. Making implementation science more rapid: Use of the RE-AIM framework for mid-course adaptations across five health services research projects in the Veterans Health Administration. Front Public Health. (2020).

Google Scholar

31. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. (2015) 10:109. doi: 10.1186/s13012-015-0295-0

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Balasubramanian BA, Cohen DJ, Davis MM, Gunn R, Dickinson LM, Miller WL, et al. Learning evaluation: blending quality improvement and implementation research methods to study healthcare innovations. Implement Sci. (2015) 10:31. doi: 10.1186/s13012-015-0219-z

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Coury J., Schneider J. L., Rivelli J. S., Petrik A. F., Seibel E., D'agostini B., et al. (2017). Applying the Plan-Do-Study-Act (PDSA) approach to a large pragmatic study involving safety net clinics. BMC Health Services Res. 17:411. doi: 10.1186/s12913-017-2364-3

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Nilsen P. Making sense of implementation theories, models and frameworks. Implementat Sci. (2015) 10:53. doi: 10.1186/s13012-015-0242-0

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Joint Commission J Qual Patient Safety. (2008) 34:228–43. doi: 10.1016/S1553-7250(08)34030-6

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4:50–50. doi: 10.1186/1748-5908-4-50

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: a new instrument for public health programs. Prevent Chronic Dis. (2014) 11:130184. doi: 10.5888/pcd11.130184

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Braveman P. A new definition of health equity to guide future efforts and measure progress. Health Affairs Blog. (2017) 22:1–4.

Google Scholar

39. Chinman M, Woodward EN, Curran GM, Hausmann LR. Harnessing implementation science to increase the impact of health disparity research. Medical Care. (2017) 55:S16. doi: 10.1097/MLR.0000000000000769

CrossRef Full Text | Google Scholar

40. Dumont K, Metz A, Woo B. Five Recommendations for How Implementation Science Can Better Advance Equity (2019). Available from: www.academyhealth.org/blog/2019-04/five-recommendations-how-implementation-science-can-better-advance-equity (accessed September 26, 2019).

Google Scholar

41. Ritzwoller DP, Sukhanova AS, Glasgow RE, Strycker LA, King DK, Gaglio B, et al. Intervention costs and cost-effectiveness for a multiple-risk-factor diabetes self-management trial for Latinas: economic analysis of ¡Viva Bien! Transl Behav Med. (2011) 1:427–35. doi: 10.1007/s13142-011-0037-z

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Jones Rhodes WC, Ritzwoller DP, Glasgow RE. Stakeholder perspectives on costs and resource expenditures: tools for addressing economic issues most relevant to patients, providers, and clinics. Transl Behav Med. (2018) 8:675–82. doi: 10.1093/tbm/ibx003

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Saldana L, Chamberlain P, Bradford WD, Campbell M, Landsverk J. The Cost of Implementing New Strategies (COINS): a method for mapping implementation resources using the stages of implementation completion. Children Youth Services Rev. (2014) 39:177–82. doi: 10.1016/j.childyouth.2013.10.006

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Dopp AR, Mundey P, Beasley LO, Silovsky JF, Eisenberg D. Mixed-method approaches to strengthen economic evaluations in implementation research. Implement Sci. (2019) 14:2. doi: 10.1186/s13012-018-0850-6

PubMed Abstract | CrossRef Full Text | Google Scholar

45. May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. (2016) 11:141. doi: 10.1186/s13012-016-0506-3

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Braithwaite J, Churruca K, Long JC, Ellis LA, Herkes J. When complexity science meets implementation science: a theoretical and empirical analysis of systems change. BMC Med. (2018) 16:63. doi: 10.1186/s12916-018-1057-z

PubMed Abstract | CrossRef Full Text | Google Scholar