A commentary has been posted on this article:
Commentary: RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review
Russell E. Glasgow1*
Samantha M. Harden2
Bridget Gaglio3
Borsika Rabin1,4
Matthew Lee Smith5,6,7
Gwenndolyn C. Porter8
Marcia G. Ory5,6Paul A. Estabrooks8- 1Dissemination and Implementation Science Program of ACCORDS, Department of Family Medicine, School of Medicine, University of Colorado, Aurora, CO, United States
- 2Physical Activity Research and Community Implementation, Human Nutrition, Foods, and Exercise, Virginia Tech, Blacksburg, VA, United States
- 3Patient-Centered Outcomes Research Institute, Washington, DC, United States
- 4Department of Family Medicine and Public Health, School of Medicine, University of California, San Diego, La Jolla, CA, United States
- 5Center for Population Health and Aging, Texas A&M University, College Station, TX, United States
- 6Department of Environmental and Occupational Health, School of Public Health, Texas A&M University, College Station, TX, United States
- 7Department of Health Promotion and Behavior, College of Public Health, The University of Georgia, Athens, GA, United States
- 8Department of Health Promotion, College of Public Health, University of Nebraska Medical Center, Omaha, NE, United States
The RE-AIM planning and evaluation framework was conceptualized two decades ago. As one of the most frequently applied implementation frameworks, RE-AIM has now been cited in over 2,800 publications. This paper describes the application and evolution of RE-AIM as well as lessons learned from its use. RE-AIM has been applied most often in public health and health behavior change research, but increasingly in more diverse content areas and within clinical, community, and corporate settings. We discuss challenges of using RE-AIM while encouraging a more pragmatic use of key dimensions rather than comprehensive applications of all elements. Current foci of RE-AIM include increasing the emphasis on cost and adaptations to programs and expanding the use of qualitative methods to understand “how” and “why” results came about. The framework will continue to evolve to focus on contextual and explanatory factors related to RE-AIM outcomes, package RE-AIM for use by non-researchers, and integrate RE-AIM with other pragmatic and reporting frameworks.
Introduction
The RE-AIM framework (1) was developed to address the issue that the translation of scientific advances into practice, and especially into public health impact and policy, have been slow and inequitable (2–6). RE-AIM and other models (7) have helped balance the traditional focus on internal over external validity. Unique features of RE-AIM include an explicit focus on issues, dimensions, and steps in the design, dissemination, and implementation process that can either facilitate or impede success in achieving broad and equitable population-based impact.
The seminal RE-AIM paper (1) has been cited over 2,800 times, and the RE-AIM framework has been applied to study planning or evaluation in over 450 publications (7). RE-AIM is one of the most frequently used frameworks for planning and evaluation of grant applications at most of the leading U.S. health and medical research agencies (8) and has been used widely (nationally and internationally) (9, 10) and across populations, settings, and health conditions (11–24). Generally, RE-AIM does seem to translate and be useful in the different countries and cultures in which use has been reported. Some international applications include low- and middle-income countries (25) including Australia (26–30), the Netherlands (31–34), and Brazil (35, 36). One interesting application was the use of RE-AIM to help plan and evaluate interventions to reduce the use of coal-fired indoor cook stoves in Africa (37). In this article, we summarize the history of the RE-AIM framework, discusses current applications of RE-AIM for research and practice, and outline opportunities for future application.
Historical Perspectives
The dimensions of the RE-AIM framework were originally introduced to encourage scientists to be more transparent and consider internal and external validity across pilot, efficacy, effectiveness, demonstration, and translational research (23, 38). Most peer reviewed publications previously emphasized efficacy, leaving researchers, and practitioners with little information about the generalizability of the intervention context, implementation personnel and conditions, and findings. The main goal from its conception was to improve assessment and reporting along the five RE-AIM dimensions, not necessarily intervening to improve all dimensions (see Table 1).
TABLE 1
Table 1. The RE-AIM dimensions: definitions, evolution, and examples from the literature.
The RE-AIM dimensions include reach (R), effectiveness (E), and maintenance (M)–which operate at the individual-level (i.e., those who are intended to benefit), and adoption (A), implementation (I), and maintenance (M), which focus on the staff and setting levels. Setting-level RE-AIM factors are often multi-level and address context and external validity issues important to population impact. For example, settings may include clinics, schools, or worksites nested within communities or larger systems, and within these settings are nested clinicians, teachers, or human resources staff responsible for implementation.
All RE-AIM dimensions are complex, but implementation currently has the most indices. It focuses on fidelity to an intervention: the extent to which the program is implemented consistently across different settings, staff, and patients. It also includes adaptations made (58) and costs from multiple stakeholder perspectives (59). Maintenance has indices at the individual- (long-term effectiveness) and setting-level (sustainability after original research funded is completed).
The framework's operational components have been increasingly applied over the years. For example, in the past, studies reported participant characteristics that differed between study conditions or between those retained and those lost to follow-up. However, studies using RE-AIM compared the representativeness of individuals who enrolled in a study to the characteristics of the intended population. These comparisons used in RE-AIM studies increased understanding about access, awareness, appropriateness, and likely generalizability of recruitment strategies and intervention approaches.
In the past, clinical effectiveness research focused relatively narrowly on physiologic outcomes. RE-AIM expanded this focus to multiple factors that impact public health. This approach to assessing broader impacts aided in understanding comprehensive effects of a program on quality of life, including unintended consequences (e.g., increasing health inequity or the social stigma of labeling someone with a chronic condition).
There have been several literature reviews on use of RE-AIM (11, 42, 52, 60–62). The most comprehensive reviews have spanned the literature from 2000 to 2012 or 2015 (10, 42, 52). Notably, these reviews of different content areas reached similar conclusions: that adoption and maintenance, as well as representativeness across individual- and organizational-levels, were reported far less frequently. They identified frequent issues with confusing different dimensions, in particular reach (at the individual-level) and adoption (at the setting-level). These observations are not limited to the United States alone.
To enhance development and application of the framework, several scientists contributed to a RE-AIM research consortium funded by the Robert Wood Johnson Foundation (63–66). This work led to the development of a website, www.re-aim.org, in 2004 (64). The website serves as a repository of various resources and tools including self-quizzes, checklists, figures, tables, measures, tips for using RE-AIM, and increasingly, other social media tools. These are available to facilitate the operationalization and application of RE-AIM across diverse interventions, settings, and populations. To enhance a dialogue within the broader research community, monthly webinars are held about RE-AIM related issues; archived recordings are available on the website (www.re-aim.org).
From Past to Present
Below, we summarize five general areas currently being examined using RE-AIM (Table 1). The first is to understand and maximize the potential of RE-AIM to assess adaptations prior to, during, and after program implementation (54, 55, 67). Adaptations naturally occur during the implementation of programs (68). Mittman et al. (69) suggest that instead of ignoring or suppressing this phenomenon, we should find ways to document and assess these changes. Recent Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards (70) suggest that adaptations should be systematically documented. RE-AIM has great potential to provide guidance about documenting adaptations. It can also provide guidance about how to evaluate the impact of these adaptations, as well as their purpose (58). RE-AIM has been used to expand the widely known Stirman framework (71) on adaptations with additional components to address “who, what, why, where, and when” questions (67). RE-AIM considers adaptations in a longitudinal, multi-method, and multi-level manner and includes data collection at multiple time points and from multiple stakeholders, using multiple data collection approaches (54, 55, 67).
Second, there has been a recent focus on more qualitative RE-AIM assessments. Most evaluations and uses of the framework have emphasized descriptive or quantitative data, often focusing on key aspects such as the percentage of potentially eligible persons or settings that participate. A qualitative focus as presented by Holtrop et al. (72) can enhance understanding about what happened as well as the “how” and the “why.”
Third, we recognized the need for more pragmatic uses of the framework rather than trying to comprehensively assess all RE-AIM dimensions in all applications, especially when not having many evaluation resources (67, 73). All studies or evaluations, and particularly those without large evaluation budgets, do not need to assess all components of RE-AIM. Rather, they should address those components most valued and appropriate for their particular question, setting, stakeholders, and stage of research. An a priori decision should be made, however, to select the dimensions on which to focus for evaluation and on which to use for planning and improvement (i.e., beyond the evaluation scope). In some cases, the decision to capture all five dimensions is made a priori to understand individual impacts, contextual implications, and feasibility of ongoing data collection. This is demonstrated in two recent applications of RE-AIM before, during, and after program implementation—and on limited funds (74, 75). Both applications highlighted the need for stakeholder buy-in (54) and operationalization (67) of each dimension that holds value for these stakeholders.
Fourth, assessment of costs, from the perspective of multiple stakeholders and across the various RE-AIM dimensions, is another area of emphasis (5). Building upon earlier work by Ritzwoller et al. (76), recent RE-AIM cost assessments have focused on the multilevel nature of implementation, different stakeholder perspectives, and cost estimates for replicating a program or policy in different settings. Rhodes et al. (59) have provided templates to assess costs at the patient-, staff-, clinic-, and organizational-levels. Costs to deliver programs are associated with activities to address and enhance each RE-AIM dimension (Figure 1). In the future, we anticipate more consistent reporting of costs and burden and more frequent comparative effectiveness research about cost-effective methods to enhance value and various RE-AIM dimensions.
FIGURE 1
Figure 1. Revised, enhanced RE-AIM/PRISM 2019 model.
Finally, Glasgow et al. (77) have recently advocated for an extension of RE-AIM concepts and dimensions, termed an Expanded CONSORT Figure to enhance transparent reporting, and potentially, replication. The goal is to expand the CONSORT reporting criteria required for randomized studies (78) to (a) include factors related to setting and staff level participation and representativeness, which begin before individual participants are recruited, and (b) extend the temporal focus beyond the end of a study. The expanded CONSORT figure and a related downloadable template summarize issues of exclusion and inclusion criteria for settings (e.g., communities or healthcare networks) and delivery staff, [e.g., evaluating the percent and characteristics of settings and staff that participate or do not (adoption)], reasons for participation or non-participation, and intervention sustainability after project support ends (79).
Based on these observations, we have developed a new RE-AIM figure to highlight the various changes to the model, as well as new emphases, including explicit inclusion of costs and adaptations, as shown in Figure 1. The figure also emphasizes key multi-level contextual factors (both the internal and external context) that influence RE-AIM outcomes as discussed below. Two crosscutting issues are: (a) that it is critical that there is alignment across setting and context, the intervention and implementation strategies; and (b) it is important to include qualitative assessments to determine how and why various RE-AIM outcomes are produced.
What Will the Future Bring?
With the historical context and current application of RE-AIM in mind, we outline five future directions for researchers and practitioners interested in using RE-AIM.
First, there have been recent calls to more explicitly describe strategies and context (80, 81) as well as test mediating relationships between implementation strategies and implementation outcomes (82). We also see this as an emerging area for RE-AIM. The most well-articulated attempt to do this so far is the Practical, Robust, Implementation, and Sustainability Model (PRISM) (83, 84) that focuses on specific contextual factors from external macro-level factors such as policies, guidelines, and incentives, to more local organizational-level factors. It focuses on the fit between the characteristics of an intervention (i.e., Rogers' constructs of relative advantage, complexity, compatibility, observability, trialability, and cost) (85) and the particular intervention and implementation system. A somewhat unique factor of PRISM is its focus on enhancing setting-level maintenance characteristics by addressing the “implementation and sustainability infrastructure”—including job requirements, ongoing audit and feedback, and institutionalization of intervention activities (84, 86).
Second, mixed-methods should be used across framework components to identify explanatory processes across RE-AIM dimensions. To date, quantitative measures alone have been insufficient to strongly predict dissemination (reach and adoption), implementation, and maintenance outcomes. Using mixed-methods approaches can help identify factors that are causally related to different RE-AIM outcomes in different situations (72). Qualitative information integrated with newer predictive modeling approaches should provide more detailed guidance about actions that can be taken to enhance outcomes by addressing empirically-derived causal relationships.
Third, we encourage more iterative applications of RE-AIM and use of the framework during the implementation period, not just for initial planning and summative evaluation. Rapid, iterative use and analysis of brief practical measures of RE-AIM factors can inform adaptations (55, 66, 71). In brief, RE-AIM can be used as part of a participatory approach (Estabrooks et al., under review), to determine which dimensions should be assessed, described, or targeted for intervention. For example, a recruitment strategy may need to be adapted over the life-course of an intervention (to improve reach) or a new training strategy may be employed (to improve adoption and implementation). While often examined and interpreted independently, these adaptations can work together to be empirically robust and practically meaningful.
Another issue to be addressed is use of RE-AIM by non-researchers and groups such as state health programs or program evaluators without substantial funds (grants/contracts). Using RE-AIM in low-resource and real-world settings can be challenging but successful (2). Preliminary findings assessing such use are that RE-AIM is used widely, and seems to be relatively intuitive, but there are challenges implementing it at a detailed level and assessing all components. The development of user-friendly tools and aids using human centered design, as well as more examples of the application of RE-AIM for such users is an important future direction.
Finally, we think there is great opportunity for RE-AIM to be used in combination with other approaches such as the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) model (87, 88), where RE-AIM factors can be combined with the PRECIS-2 dimensions to determine how pragmatic a study is and how generalizable it is likely to be. Such use is illustrated in a recent systematic review by Luoma et al. (89), who demonstrated how reviews can simultaneously summarize effectiveness (using Cochrane-type criteria) and pragmatism (using a combination of PRECIS-2 and RE-AIM factors). RE-AIM and its Expanded CONSORT extension could also be integrated with the Standards for Reporting Implementation Studies (StaRI) (90) or other dissemination and implementation (D&I) research reporting criteria.
Conclusion
RE-AIM has been applied in research and practice for 20 years. Although its original components have remained, much has been modified and evolved to address emerging issues such as adaptation and dissemination costs. We expect that RE-AIM will continue to evolve to better address and enhance its original purpose—to increase the prevalence of relevant research that can be applied broadly across a wide variety of populations and settings to achieve a large, equitable, and replicable public health impact. We invite researchers and practitioners to contribute to the expanded use of RE-AIM before, during, and after intervention delivery.
Author Contributions
All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding
The authors would like to acknowledge funding support for author contributions: MO and MS contributions supported through ACL SUSTAIN for Better Health and Health Care for Older Adults 90CS0065-01. PE contributions supported by Great Plains IDEA CTR U54 GM115458- 01. GP contributions supported by American Heart Association 18PRE34060136. RG contributions partially supported by IMPlementation to Achieve Clinical Transformation (IMPACT): the Colorado Training Program from the NIH K12 HL137862, and the Eastern Colorado VA Geriatric Research Education and Clinical Center.
Disclaimer
All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The authors would like to acknowledge all members of the National Working Group on RE-AIM Planning and Evaluation Framework (www.re-aim.org). A brief You Tube video related to the content in this paper is available at: https://www.youtube.com/watch?v=thpAniodvQU&t=3s&list=PLIM7iq2xw8VwTlvd7wVud0ET-uAbfT8hf&index=2.
References
1. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/AJPH.89.9.1322
2. Glasgow RE, Askew S, Purcell P, Levine E, Warner ET, Stange KC, et al. Use of RE-AIM to address health inequities: application in a low-income community health center-based weight loss and hypertension self-management program. Transl Behav Med. (2013) 3:200–10. doi: 10.1007/s13142-013-0201-8
3. Gaglio B, Phillips SM, Heurtin-Roberts S, Sanchez MA, Glasgow RE. How pragmatic is it? lessons learned using PRECIS and RE-AIM for determining pragmatic characteristics of research. Implement Sci. (2014) 9:96. doi: 10.1186/s13012-014-0096-x
4. Glasgow RE, Nelson CC, Strycker LA, King DK. Using RE-AIM metrics to evaluate diabetes self-management support interventions. Am J Prev Med. (2006) 30:67–73. doi: 10.1016/j.amepre.2005.08.037
5. Estabrooks PA, Allen KC. Updating, employing, and adapting: a commentary on what does it mean to “employ” the RE-AIM model. Eval Health Prof. (2013) 36:67–72. doi: 10.1177/0163278712460546
6. Estabrooks PA, Glasgow RE. Translating effective clinic-based physical activity interventions into practice. Am J Prev Med. (2006) 31 (4 Suppl.):S45–56. doi: 10.1016/j.amepre.2006.06.019
7. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. (2012) 43:337–50. doi: 10.1016/j.amepre.2012.05.024
8. Vinson C, Stamatakis K, Kerner J. Dissemination and implementation research in community and public health settings. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health; Translating Research to Practice. Oxford, NY: Oxford University Press (2018). p. 355–70.
9. Shoup JA, Gaglio B, Varda D, Glasgow RE. Network analysis of RE-AIM framework: chronology of the field and the connectivity of its contributors. Transl Behav Med. (2015) 5:216–32. doi: 10.1007/s13142-014-0300-1
10. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Heal Prof. (2013) 36:44–66. doi: 10.1177/0163278712446066
11. Galaviz KI, Harden SM, Smith E, Blackman KC, Berrey LM, Mama SK, et al. Physical activity promotion in Latin American populations: a systematic review on issues of internal and external validity. Int J Behav Nutr Phys Act. (2014) 11:77. doi: 10.1186/1479-5868-11-77
12. Kim AE, Towers A, Renaud J, Zhu J, Shea JA, Galvin R, et al. Application of the RE-AIM framework to evaluate the impact of a worksite-based financial incentive intervention for smoking cessation. J Occup Environ Med. (2012) 54:610–4. doi: 10.1097/JOM.0b013e31824b2171
13. Chen I, Money D, Yong P, Williams C, Allaire C. An evaluation model for a multidisciplinary chronic pelvic pain clinic: application of the RE-AIM framework. J Obstet Gynaecol Can. (2015) 37:804–9. doi: 10.1016/S1701-2163(15)30151-1
14. Ory MG, Altpeter M, Belza B, Helduser J, Zhang C, Smith ML. Perceived utility of the RE-AIM framework for health promotion/disease prevention initiatives for older adults: a case study from the U.S. evidence-based disease prevention initiative. Front Public Health. (2015) 2:143. doi: 10.3389/fpubh.2014.00143
15. Ory MG, Altpeter M, Belza B, Helduser J, Zhang C, Smith ML. Perceptions about community applications of RE-AIM in the promotion of evidence-based programs for older adults. Eval Heal Prof. (2015) 38:15–20. doi: 10.1177/0163278714542335
16. Aerts I, Cumps E, Verhagen E, Mathieu N, Van Schuerbeeck S, Meeusen R. A 3-month jump-landing training program: a feasibility study using the RE-AIM framework. J Athl Train. (2013) 48:296–305. doi: 10.4085/1062-6050-48.3.18
17. Altpeter M, Gwyther LP, Kennedy SR, Patterson TR, Derence K. From evidence to practice: using the RE-AIM framework to adapt the REACHII caregiver intervention to the community. Dementia. (2015) 14:104–13. doi: 10.1177/1471301213494499
18. Schwingel A, Gálvez P, Linares D, Sebastião E. Using a mixed-methods RE-AIM framework to evaluate community health programs for older latinas. J Aging Health. (2017) 29:551–93. doi: 10.1177/0898264316641075
19. Adams EJ, Chalkley AE, Esliger DW, Sherar LB. Evaluation of the implementation of a whole-workplace walking programme using the RE-AIM framework. BMC Public Health. (2017) 17:466. doi: 10.1186/s12889-017-4376-7
20. Brace AM, Padilla HM, DeJoy DM, Wilson MG, Vandenberg RJ, Davis M. Applying RE-AIM to the evaluation of FUEL your life : a worksite translation of DPP. Health Promot Pract. (2015) 16:28–35. doi: 10.1177/1524839914539329
21. Ward S, Chow AF, Humbert ML, Bélanger M, Muhajarine N, Vatanparast H, et al. Promoting physical activity, healthy eating and gross motor skills development among preschoolers attending childcare centers: process evaluation of the Healthy Start-Départ Santé intervention using the RE-AIM framework. Eval Program Plann. (2018) 68:90–8. doi: 10.1016/j.evalprogplan.2018.02.005
22. Glasgow RE, Gaglio B, Estabrooks PA, Marcus AC, Ritzwoller DP, Smith TL, et al. Long-term results of a smoking reduction program. Med Care. (2009) 47:115–20. doi: 10.1097/MLR.0b013e31817e18d1
23. Dzewaltowski DA, Estabrooks PA, Glasgow RE. The future of physical activity behavior change research: what is needed to improve translation of research into health promotion practice? Exerc Sport Sci Rev. (2004) 32:57–63. doi: 10.1097/00003677-200404000-00004
24. Estabrooks P, Dzewaltowski DA, Glasgow RE, Klesges LM. Reporting of validity from school health promotion studies published in 12 leading journals, 1996-2000. J Sch Health. (2003) 73:21–8. doi: 10.1111/j.1746-1561.2003.tb06554.x
25. Lee RE, Galavíz KI, Soltero EG, Rosales Chavez J, Jauregui E, Lévesque L, et al. Applying the RE-AIM conceptual framework for the promotion of physical activity in low-and middle-income countries. Rev Lat Am Enfermagem. (2017) 25:e2923. doi: 10.1590/1518-8345.1894.2923
26. Caperchione CM, Duncan M, Kolt GS, Vandelanotte C, Rosenkranz RR, Maeder A, et al. Examining an Australian physical activity and nutrition intervention using RE-AIM. Health Promot Int. (2015) 31:450–8. doi: 10.1093/heapro/dav005
27. O'Brien J, Finch CF. The implementation of musculoskeletal injury-prevention exercise programmes in team ball sports: a systematic review employing the RE-AIM framework. Sports Med. (2014) 44:1305–18. doi: 10.1007/s40279-014-0208-4
28. Koorts H, Gillison F. Mixed method evaluation of a community-based physical activity program using the RE-AIM framework: practical application in a real-world setting. BMC Public Health. (2015) 15:1102. doi: 10.1186/s12889-015-2466-y
29. Jenkinson KA, Naughton G, Benson AC. The GLAMA (Girls! Lead! Achieve! Mentor! Activate!) physical activity and peer leadership intervention pilot project: a process evaluation using the RE-AIM framework. BMC Public Health. (2012) 12:55. doi: 10.1186/1471-2458-12-55
30. Oldroyd JC, White S, Stephens M, Neil AA, Nanayakkara V. Program evaluation of the inner south community health oral health program for priority populations. J Health Care Poor Underserv. (2017) 28:1222–39. doi: 10.1353/hpu.2017.0107
31. Leenaars KE, Smit E, Wagemakers A, Molleman GR, Koelen MA. Exploring the impact of the care sport connector in the Netherlands. BMC Public Health. (2017) 17:813. doi: 10.1186/s12889-017-4830-6
32. Schaap FD, Dijkstra GJ, Finnema EJ, Reijneveld SA. The first use of dementia care mapping in the care for older people with intellectual disability: a process analysis according to the RE-AIM framework. Aging Ment Health. (2018) 22:912–9. doi: 10.1080/13607863.2017.1401582
33. Spook JE, Paulussen T, Paulissen R, Visschedijk G, Kok G, van Empelen P. Design rationale behind the serious self-regulation game intervention “balance it”: overweight prevention among secondary vocational education students in the Netherlands. Games Health J. (2015) 4:387–400. doi: 10.1089/g4h.2014.0142
34. Boersma P, Van Weert JC, van Meijel B, van de Ven PM, Dröes RM. Study protocol Implementation of the Veder contact method (VCM) in daily nursing home care for people with dementia: an evaluation based on the RE-AIM framework. Aging Ment Health. (2017) 21:730–41. doi: 10.1080/13607863.2016.1154015
35. Baba CT, Oliveira IM, Silva AE, Vieira LM, Cerri NC, Florindo AA, et al. Evaluating the impact of a walking program in a disadvantaged area: using the RE-AIM framework by mixed methods. BMC Public Health. (2017) 17:709. doi: 10.1186/s12889-017-4698-5
36. Almeida FA, Almeida Brito F. Planning and evaluating health programs: contributions of the RE-AIM framework to Nursing. Rev Lat Am Enfermagem. (2014) 22:527–8. doi: 10.1590/0104-1169.0000.2447
37. Quinn AK, Bruce N, Puzzolo E, Dickinson K, Sturke R, Jack DW, et al. An analysis of efforts to scale up clean household energy for cooking around the world. Energy Sustain Dev. (2018) 46:1–0. doi: 10.1016/j.esd.2018.06.011
38. Estabrooks PA, Gyurcsik NC. Evaluating the impact of behavioral interventions that target physical activity: issues of generalizability and public health. Psychol Sport Exerc. (2003) 4:41–55. doi: 10.1016/S1469-0292(02)00016-X
39. Harden SM, Fanning J, Motl RW, McAuley E, Estabrooks PA. Determining the reach of a home-based physical activity program for older adults within the context of a randomized controlled trial. Health Educ Res. (2014) 29:861–9. doi: 10.1093/her/cyu049
40. You W, Almeida FA, Zoellner JM, Hill JL, Pinard CA, Allen KC, et al. Who participates in internet-based worksite weight loss programs? BMC Public Health. (2011) 11:709. doi: 10.1186/1471-2458-11-709
41. Lee RE, Reese-Smith JY, Mama SK, Medina AV, Wolfe KL, Estabrooks PA. Reach and representativeness of ethnic minority women in the health is power study: a longitudinal analysis. Transl Behav Med. (2017) 7:106–14. doi: 10.1007/s13142-016-0385-9
42. Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB, Brito F, et al. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev. (2015) 4:155. doi: 10.1186/s13643-015-0141-0
43. Perry RA, Golley RK, Hartley J, Magarey AM. The adaptation and translation of the PEACHTM RCT intervention: the process and outcomes of the PEACHTM in the community trial. Public Health. (2017) 153:154–62. doi: 10.1016/j.puhe.2017.08.009
44. Jaipakdee J, Jiamjarasrangsi W, Lohsoonthorn V, Lertmaharit S. Effectiveness of a self-management support program for Thais with type 2 diabetes: evaluation according to the RE-AIM framework. Nurs Health Sci. (2015) 17:362–9. doi: 10.1111/nhs.12198
45. Smith SN, Almirall D, Prenovost K, Goodrich DE, Abraham KM, Liebrecht C, et al. Organizational culture and climate as moderators of enhanced outreach for persons with serious mental illness: results from a cluster-randomized trial of adaptive implementation strategies. Implement Sci. (2018) 13:93. doi: 10.1186/s13012-018-0787-9
46. Dunton GF, Liao Y, Grana R, Lagloire R, Riggs N, Chou CP, et al. State-wide dissemination of a school-based nutrition education programme: a RE-AIM (Reach, Efficacy, Adoption, Implementation, Maintenance) analysis. Public Health Nutr. (2014) 17:422–30. doi: 10.1017/S1368980012005186
47. Maxwell AE, Jo AM, Chin SY, Lee KS, Bastani R. Impact of a print intervention to increase annual mammography screening among Korean American women enrolled in the National Breast and Cervical Cancer Early Detection Program. Cancer Detect Prev. (2008) 32:229–35. doi: 10.1016/j.cdp.2008.04.003
48. Breitenstein SM, Fogg L, Garvey C, Hill C, Resnick B, Gross D. Measuring implementation fidelity in a community-based parenting intervention. Nurs Res. (2010) 59:158–65. doi: 10.1097/NNR.0b013e3181dbb2e2
49. Duffy SA, Ronis DL, Ewing LA, Waltje AH, Hall S V, Thomas PL, et al. Implementation of the Tobacco Tactics intervention versus usual care in Trinity Health community hospitals. Implement Sci. (2016) 11:147. doi: 10.1186/s13012-016-0511-6
50. Santos SLZ, Tagai EK, Scheirer MA, Bowie J, Haider M, Slade J, et al. Adoption, reach, and implementation of a cancer education intervention in African American churches. Implement Sci. (2017) 12:36. doi: 10.1186/s13012-017-0566-z
51. Estabrooks PA, Wilson KE, McGuire TJ, Harden SM, Ramalingam N, Schoepke L, et al. A quasi-experiment to assess the impact of a scalable, community-based weight loss program: combining reach, effectiveness, and cost. J Gen Intern Med. (2017) 32:24–31. doi: 10.1007/s11606-016-3971-0
52. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. (2013) 103:e38–46. doi: 10.2105/AJPH.2013.301299
53. Glasgow RE, Vinson C, Chambers D, Khoury MJ, Kaplan RM, Hunter C. National Institutes of health approaches to dissemination and implementation science: current and future directions. Am J Public Health. (2012) 102:1274–81. doi: 10.2105/AJPH.2012.300755
54. Sweet SN, Ginis KAM, Estabrooks PA, Latimer-Cheung AE. Operationalizing the RE-AIM framework to evaluate the impact of multi-sector partnerships. Implement Sci. (2014) 9:74. doi: 10.1186/1748-5908-9-74
55. Harden SM, Smith ML, Ory MG, Smith-Ray RL, Estabrooks P, Glasgow R. RE-AIM in clinical, community, and corporate settings: perspectives, strategies, and recommendations to enhance public health impact. Front Public Health. (2018) 6:71. doi: 10.3389/fpubh.2018.00071
56. Fortney J, Enderle M, McDougall S, Clothier J, Otero J, Altman L, et al. Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implement Sci. (2012) 7:30. doi: 10.1186/1748-5908-7-30
57. Carlfjord S, Lindberg M, Andersson A. Sustained use of a tool for lifestyle intervention implemented in primary health care: a 2-year follow-up. J Eval Clin Pract. (2013) 19:327–34. doi: 10.1111/j.1365-2753.2012.01827.x
58. Rabin BA, McCreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, multimethod assessment of adaptations across four diverse health systems interventions. Front Public Health. (2018) 6:102. doi: 10.3389/fpubh.2018.00102
59. Jones Rhodes WC, Ritzwoller DP, Glasgow RE. Stakeholder perspectives on costs and resource expenditures: tools for addressing economic issues most relevant to patients, providers, and clinics. Transl Behav Med. (2018) 8:675–82. doi: 10.1093/tbm/ibx003
60. Akers JD, Estabrooks PA, Davy BM. Translational research: bridging the gap between long-term weight loss maintenance research and practice. J Am Diet Assoc. (2010) 110:1511–22, 1522-3. doi: 10.1016/j.jada.2010.07.005
61. Allen K, Zoellner J, Motley M, Estabrooks PA. Understanding the internal and external validity of health literacy interventions: a systematic literature review using the RE-AIM framework. J Health Commun. (2011) 16 (Suppl. 3):55–72. doi: 10.1080/10810730.2011.604381
62. Bull SS, Gillette C, Glasgow RE, Estabrooks P. Work site health promotion research: to what extent can we generalize the results and what is needed to translate research to practice? Heal Educ Behav. (2003) 30:537–49. doi: 10.1177/1090198103254340
63. Klesges LM, Estabrooks PA, Dzewaltowski DA, Bull SS, Glasgow RE. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med. (2005) 29:66–75. doi: 10.1207/s15324796abm2902s_10
64. Dzewaltowski DA, Glasgow RE, Klesges LM, Estabrooks PA, Brock E. RE-AIM: evidence-based standards and a Web resource to improve translation of research into practice. Ann Behav Med. (2004) 28:75–80. doi: 10.1207/s15324796abm2802_1
65. Ory MG, Jordan PJ, Bazzarre T. The behavior change consortium: setting the stage for a new century of health behavior-change research. Health Educ Res. (2002) 17:500–11. doi: 10.1093/her/17.5.500
66. Glasgow RE, Bull SS, Gillette C, Klesges LM, Dzewaltowski DA. Behavior change intervention research in healthcare settings: a review of recent reports with emphasis on external validity. Am J Prev Med. (2002) 23:62–9. doi: 10.1016/S0749-3797(02)00437-3
67. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. (2018) 15:170271. doi: 10.5888/pcd15.170271
68. Chambers DA, Norton WE. The adaptome: advancing the science of intervention adaptation. Am J Prev Med. (2016) 51:S124–31. doi: 10.1016/j.amepre.2016.05.011
69. Mittman B. Evaluating Complex Interventions: Confronting and Guiding (Versus Ignoring and Suppressing) Heterogeneity and Adaptation. PSMG Northwestern Grand Rounds webinar series. (2018) Available online at: http://cepim.northwestern.edu/psmg-archive/ (Accessed December 12, 2018).
70. Patient-Centered Outcomes Research Institute (PCORI). Methodology Standards. (2019) Available online at: https://www.pcori.org/sites/default/files/PCORI-Methodology-Standards.pdf
71. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. (2013) 8:65. doi: 10.1186/1748-5908-8-65
72. Holtrop JS, Rabin BA, Glasgow RE. Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res. (2018) 18:177. doi: 10.1186/s12913-018-2938-8
73. Stoutenberg M, Galaviz KI, Lobelo F, Joy E, Heath GW, Hutber A, et al. A pragmatic application of the RE-AIM framework for evaluating the implementation of physical activity as a standard of care in health systems. Prev Chronic Dis. (2018) 15:E54. doi: 10.5888/pcd15.170344
74. Balis LE, Strayer III TE, Ramalingam N, Harden SM. Beginning with the end in mind: contextual considerations for scaling-out a community-based intervention. Front Public Health. (2018) 6:357. doi: 10.3389/fpubh.2018.00357
75. Harden SM, Ramalingam NS, Breig SA, Estabrooks PA. Walk this way: our perspective on challenges and opportunities for extension statewide walking promotion programs. J Nutr Educ Behav. (2019). doi: 10.1016/j.jneb.2018.12.010. [Epub ahead of print].
76. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide to enhance translation. Ann Behav Med. (2009) 37:218–27. doi: 10.1007/s12160-009-9088-5
77. Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT figure: increasing transparency in reporting on external validity. Am J Prev Med. (2018) 55:422–30. doi: 10.1016/j.amepre.2018.04.044
78. Moher D. CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. JAMA. (1998) 279:1489–91. doi: 10.1001/jama.279.18.1489
79. Glasgow R, Huebschmann A, Brownson R. Expanded CONSORT Figure for Planning and Reporting D & I Research. (2018) Available online at: http://www.re-aim.org/expanded-consort-figure-for-planning-and-reporting-d-i-research/ (Accessed December 13, 2018).
80. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. (2011) 38:65–76. doi: 10.1007/s10488-010-0319-7
81. Proctor E, Powell B, McMillen J. Implementation strategies: recommendations for specifying and reporting. Implement Sci. (2013) 8:139. doi: 10.1186/1748-5908-8-139
82. Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. (2018) 6:136. doi: 10.3389/fpubh.2018.00136
83. Feldstein A, Glasgow R. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. (2008) 34:228–43. doi: 10.1016/S1553-7250(08)34030-6
84. Liles EG, Schneider JL, Feldstein AC, Mosen DM, Perrin N, Rosales AG, et al. Implementation challenges and successes of a population-based colorectal cancer screening program: a qualitative study of stakeholder perspectives. Implement Sci. (2015) 10:41. doi: 10.1186/s13012-015-0227-z
86. Beck A, Bergman DA, Rahm AK, Dearing JW, Glasgow RE. Using implementation and dissemination concepts to spread 21st-century well-child care at a health maintenance organization. Perm J. (2009) 13:10. doi: 10.7812/TPP/08-088
87. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. (2009) 62:464–75. doi: 10.1016/j.jclinepi.2008.12.011
88. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. (2015) 350:h2147. doi: 10.1136/bmj.h2147
89. Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG, Dunn AL, et al. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review. Transl Behav Med. (2017) 7:751–72. doi: 10.1007/s13142-017-0502-4
Keywords: RE-AIM, evaluation, external validity, dissemination, implementation
Citation: Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, Ory MG and Estabrooks PA (2019) RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front. Public Health 7:64. doi: 10.3389/fpubh.2019.00064
Received: 09 January 2019; Accepted: 05 March 2019;
Published: 29 March 2019.
Edited by:
Dan J. Graham, Colorado State University, United StatesReviewed by:
Melissa Bopp, Pennsylvania State University, United StatesDeborah Paone, Independent Researcher, United States
Copyright © 2019 Glasgow, Harden, Gaglio, Rabin, Smith, Porter, Ory and Estabrooks. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Russell E. Glasgow, russell.glasgow@ucdenver.edu