National Working Group on the RE-AIM Planning and Evaluation Framework: Goals, Resources, and Future Directions

The National Working Group on RE-AIM Planning and Evaluation Framework (herein Workgroup) was established in 2004 to support the application of the framework and advance dissemination and implementation science (D&I). Workgroup members developed and disseminated products and resources (and continue to do so) to advocate for consistent application of RE-AIM and allow for cross study comparisons. The purpose of this paper is to summarize key Workgroup activities, products, and services (e.g., webinars, consultations, planning tools) and enhance bidirectional communication between the Workgroup and RE-AIM users. The ultimate goal of this work is to serve as a forum for dissemination to improve the balance between RE-AIM user demand (needs) and the currently limited RE-AIM Workgroup supply (consultation and resources) to demonstrate and expand the utility of RE-AIM as a D&I planning and evaluation framework. A summary of resources is provided as well as specific examples of how the Workgroup has been responsive to user needs.

Introduction and History

Dissemination and implementation (D&I) research is designed to facilitate integration of research into practice and policy and study the process by which this occurs (1). D&I science includes a number of models, theories, frameworks (herein models) to guide D&I science activities that have become much more prominent in recent years (2, 3). However, researchers often struggle operationalizing these models in their work (3). Resources and guidance are needed to support both researchers and practitioners to apply D&I models to their work in a meaningful way. In response to this need, a number of training programs were developed to support immersion in D&I models, methods, and measures (4). These trainings typically combine in person and virtual meetings, mentorship, reading lists, and webinars. Evaluations of these trainings have been positive, but they have limited resources (i.e., mentors, funding, time) and therefore, limited reach among people who wish to apply D&I models in practice. To respond to these demands, several research groups provide a suite of materials to introduce and guide the use of specific D&I models (5–8). The suite of options typically showcases websites, webinars, static materials, and opportunities for consultation (7).

This suite of D&I model resources should have capabilities to reach and be useful to a large number of researchers and, of equal or greater importance, be understandable and useable for practitioners and healthcare teams. These resources should also provide user support to ensure adoption, consistent application of ideas, and, when necessary, appropriate adaptation of model implementation. This paper is centered on one of the most frequently used (9–11) D&I models, RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) (12) and the various activities conducted since 2004 to provide general and tailored guidance and support for the operationalization of RE-AIM.

RE-AIM was developed in the late 1990s with the intent to support the translation of research to practice and policy, to improve the impact of health promotion and prevention efforts (4, 9, 12–14). The RE-AIM website, initially funded by the Robert Wood Johnson Foundation, was developed in 2004 to enhance understanding and transparency of the framework, provide application resources and address commonly asked questions (8). The website has consistently included RE-AIM-relevant papers, presentations, definitions, and checklists. Over the years, interactive materials have been developed including self-quizzes with feedback and suggestions, webinar viewings, and RE-AIM calculators. With limited funding, the website has been continuously supported by RE-AIM investigators, called the National Working Group on RE-AIM Planning and Evaluation Framework (herein the Workgroup). Members of the Workgroup consist of senior investigators who have applied RE-AIM since its inception and early career researchers who studied and applied RE-AIM in graduate school. This Workgroup is similar to other research or professional networks; membership is not exclusive but rather driven by shared tasks and mission (15–17).

The purpose of this paper is to summarize key activities, products, and services (e.g., webinars and planning tools) that the Workgroup offers. By doing so, Workgroup members invite RE-AIM users to provide feedback and recommend additional resources that could contribute to advancing D&I science. Some key efforts of the Workgroup are summarized including: (1) website resources, evolution, and future directions; (2) papers, presentations, and webinars; and (3) consultation experiences and offerings (e.g., grant consultations, trainings, and practice facilitation). It is our intent to connect those applying RE-AIM in clinical, corporate, or community settings with necessary resources, and also inspire others to apply the model, share new applications, and communicate additional needs. The ultimate goal is to serve as a forum for dissemination (18) to improve the balance between RE-AIM user demand (needs) and the RE-AIM Workgroup supply (offerings) to promote usefulness of RE-AIM as a D&I planning and evaluation framework.

Goals of the Workgroup

The mission of the RE-AIM Workgroup is to implement a robust and evolving framework to advance science, enhance practice, and influence policy through collaboration and training. In addition to collaborative research to advance RE-AIM, members provide guidance for its use (particularly as related to its evolution in response to new settings, purposes, opportunities, data, and challenges) (9), In a recent summary of the RE-AIM framework over its 20 year history (9), published by the Workgroup, a number of suggestions for future use of RE-AIM were put forth. These included (1) enhanced reporting and evaluation of implementation context and strategies; (2) application of mixed-methods research designs; (3) more rapid and iterative use of RE-AIM; and (4) combining RE-AIM with other relevant D&I and pragmatic frameworks and approaches such as the Pragmatic Explanatory Continuum Indicator Summary (PRECIS-2) model (19, 20). These suggestions were, in part, in response to the needs of RE-AIM users shared with members of the Workgroup—via email, in person conversations, or website requests. By way of example, one Workgroup member recently consulted on a project reviewing physical activity self-management for patients with spinal cord injury that applied both PRECIS-2 and RE-AIM (21). This led to development of a new data extraction tool, discussions among co-authors on the nuances of the RE-AIM framework, and swift responses to peer review journal manuscript reviewer comments. To further reporting and evaluation, two members of the Workgroup completed a study to qualitatively identify and assess the planning, implementation, evaluation, and dissemination using RE-AM and the Practical Robust Implementation and Sustainability Model (PRISM) extension of the framework across four health services intervention projects in the Veterans health administration setting. PRISM is a contextual expansion of the RE-AIM framework which includes theoretical constructs hypothesized to be predictive of each of the RE-AIM dimensions. The constructs include intervention characteristics, recipient characteristics, implementation and sustainability infrastructure, and external environment domains (22, 23). Results of the Veterans health administration projects pointed to the need to engage key stakeholders, assess how an intervention “fits” the targeted system, and to adjust/adapt over time and for different settings as keys for the success of dissemination and implementation efforts (23). An ongoing effort of a sub-group of Workgroup members is using a protocol to apply RE-AIM iteratively to inform adaptations during the implementation of health services interventions across four health services research projects. This effort starts with periodic evaluative reflections by the implementation team for the project including ratings on the importance of and progress on the various RE-AIM dimensions. This exercise is followed by the selection of and goal setting around one to two RE-AIM dimensions that are most important at the given time of the project and could benefit most from improvement. Examples include the selection of and goal setting on RE-AIM dimensions of (a) “reach,” with activities identified as better specification of target audience or eligible population at each implementation site, outreach to sites to assess barriers for increasing recruitment of eligible participants; and (b) “effectiveness,” to align original implementation and clinical outcomes to better match changed organizational priorities.

Resources and Activities Supported by the Workgroup

Overview of the RE-AIM Website

The purpose of the Workgroup website (www.RE-AIM.org) (8) is to support and connect those in need of explanation or resources to apply the RE-AIM framework. Specifically, the Workgroup website introduces the framework and supports implementation researchers and practitioners with guidance on application and reporting of framework dimensions. The site contains a collection of tools as well as contextual and measurement considerations when using RE-AIM in implementation research (see Figure 1 for illustration of website homepage). For example, it has a full description of each RE-AIM dimension, the methods to conduct research and planning for each, and publications about how these methodologies are applied. Finally, the website serves as an intermediary to connect users to other implementation resources such as the National Cancer Institute's Research-Tested Intervention Programs (RTIPs) and Implementation Science websites (24).

FIGURE 1

Figure 1. http://www.RE-AIM.org website layout.

Who Is Using the Website?

Google analytics software was added to the RE-AIM site in January of 2015. Data are tracked to monitor the number of sessions started, unique users who access the website, the number of page views, average time for each session, and which pages have the most visits and downloads. From January 2015 to March of 2017, these data were reported quarterly, and since April of 2017, they have been reported monthly. In 2015, there were 3,387 unique users and 3,531 sessions initiated. The website usage has increased over time and in 2018, there were 32,793 unique users (36% increase from 2017) and 48,236 (31% increase from 2017) sessions. Since early 2017, the majority of the sessions, come from the United States (69%), followed by Australia (19%), United Kingdom (9%), Canada (9%), Netherlands (4%), and Germany (2%).

Page Visits

Most users start with the homepage (Figure 1), the “What is the RE-AIM framework?” page or the “Frequently Asked Questions” page. The next most frequently visited resources are the “Measures and Checklists” and “Applying the RE-AIM Framework” pages. Unfortunately, there are no current means to capture patterns/pathways of navigation of RE-AIM site users, though understanding this information could aid in improving the user experience.

Resources, Tools, and Guides

A variety of both static (stand-still, downloads) and interactive (listserv, webinars, planning tools) resources are provided on the site. The website includes links to other relevant D&I resources such as cfir.org (Consolidated Framework for Implementation Research website), reporting guidelines from CONSORT (25, 26), Standards for Reporting Implementation Studies (StaRI) Statement (27), and ways for site users to find evidence-based interventions [e.g., through Research Tested Intervention Programs (RTIPs)]. Another primary resource is the RE-AIM Planning and Evaluation Tool, which was iteratively developed to assist users from various sectors (e.g., research, public health, healthcare) to consider the RE-AIM dimensions and enhance implementation over time. Launched in June 2017, the updated planning tool is a portable document format (PDF) instrument organized to provide prompts for consideration related to common challenges to successful application of RE-AIM. The tool prompts users to consider to whom the initiative will appeal (Whom do you plan to reach and How do you define the intended beneficiaries?). For effectiveness, the planning tool prompts users to consider: “What might be the unintended consequences or outcomes? What has gone wrong in other similar initiatives?” An adapted version of these considerations is presented in a manuscript outlining the iterative use of RE-AIM before, during, and after intervention implementation (28). A detailed example of applying the planning tool across clinical, corporate, and community settings can be found in Harden et al. (28).

RE-AIM Publication Tracking and Repository

A graduate research assistant conducts a monthly PubMed search for “RE-AIM” so users can better understand the breadth, scope and content of RE-AIM publications. See Figure 2 for the number of publications on RE-AIM, by year. In total, the RE-AIM framework has appeared in over 500 publications since its initial publication in 1999, with vastly enhanced publication rate beginning in 2009. The website includes a search feature for users to identify papers through keyword search, such as topic or target audience (e.g., physical activity, youth).

FIGURE 2

Figure 2. Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework Peer-Review Publications—frequency by year (1999–2018).

Webinar Series

The RE-AIM Workgroup and website also host and post recorded webinars featuring speakers that discuss the use of RE-AIM in research and in practice, and address questions from the audience. Each webinar is ~1 h in duration, and usually follows the format of having a moderator introduce the topic, prepared remarks from the main speaker, reflections from other Workgroup members, and time for open audience questions and discussion. From 2015 to present, there have been 27 webinars with an average of 26 live audience attendees. One of the most popular webinars to date, was delivered in January of 2018. It was titled “Reflections from the field: RE-AIMers reflect on annual D&I meeting” (10th Annual Conference on the Science of Dissemination and Implementation in Health) with panelists sharing their top lessons learned, experiences, and tips for the annual meeting.

Consultation and Presentations

Members of the Workgroup have provided a number of tutorials and workshops on the application of RE-AIM for target outlets such as the Society of Behavioral Medicine (sbm.org) and American Public Health Association (apha.org), national non-profit and federal grantee meetings in aging and public health. Members of the Workgroup also provide educational workshops, guest lectures within the U.S. and internationally (in person or via the online platform Zoom) to a variety of audiences (e.g., undergraduate kinesiology students, graduate implementation science courses, grantee meetings, and professional forums with service organizations) and on-line technical assistance (please see website under the publications and presentations sections for examples of these efforts). While not unique to the RE-AIM framework specifically, many Workgroup members are included as co-investigators or consultants on grant proposals and funded projects to bolster the application of RE-AIM. An early example was the “adoption” of the RE-AIM framework by the Centers for Disease Control and Prevention Healthy Aging Research Network as a way of understanding the impact of promoting physical activity programs for older adults (29). Workgroup members have advised national funding agencies on the use of the RE-AIM framework leading to its recommended or required inclusion in the grant applications as well as technical assistance to grantees in its practical application (30).

Publications

Complementing the more than 500 publications referencing the RE-AIM framework, core Workgroup members have recently published two key collaborative summary papers. One paper, describes the past, present, and future application of the RE-AIM framework (9). The other describes the iterative application of RE-AIM in clinical, corporate, and community settings (28). Workgroup members have also collaborated on the application of RE-AIM in a multitude of research projects and publications and provided crosscutting resources including: (1) several literature reviews of use of RE-AIM in different content areas and settings (10, 13, 31, 32); (2) how to operationalize RE-AIM in pragmatic and more practitioner-friendly language of who, what, where, when, why, and how (13); (3) use of RE-AIM to evaluate statewide walking programs in Extension (33); and (4) editing a RE-AIM-based Research Topic in the journal Frontiers in Public Health.

This wide array of publications demonstrates the utility of the framework to address scientific questions across a variety of dissemination and implementation outcomes. Specifically, the framework has been used to guide the development and assessment of interventions that expand beyond simply improving effectiveness—to include an explicit focus on improving individual and organizational-level dissemination (34, 35) of evidence-based approaches (i.e., reach and adoption, respectively) and improving implementation quality, costs, and likelihood of organizational sustainability (36, 37). It is also possible to categorize publications by different levels on a translational research spectrum from efficacy (38), to effectiveness (39), to dissemination (35), to sustainability (40) across a variety of intervention types—program, policy, systems, and environmental changes (41–43). Indeed, the accumulation of literature demonstrating the utility of the framework matches the promise to improve planning, evaluation, and scientific advancement in health promotion of early RE-AIM articles (44–46).

Future Directions

Members of the National Working Group on RE-AIM Planning and Evaluation Framework are committed to advancing D&I science through the rigorous application of the framework and related approaches across a wide variety of research areas. This involves continuously evaluating the framework's utility for planning and assessing different interventions and implementation strategies, and in diverse populations and settings. We are also continuing to push the boundaries of the framework and test its scope of applicability, and are open to making modifications to address evolving issues (9). Such D&I work is notably complex, but critical to understand a program or policy's ability to be easily adopted and adapted, reach those most in need, ensure that it is delivered with fidelity, and that it can produce replicable and long-lasting individual and systems-level improvements.

Shoup and colleagues' network analysis of RE-AIM framework use commented that researchers publishing on RE-AIM were part of an “invisible college” (47). Since 2004, members of this well-connected RE-AIM college have worked to ensure that RE-AIM use is not restricted to a small set of individuals, but instead focused on sharing resources, experiences, and novel applications of the framework. The RE-AIM website and its offerings (webinars, consultations, tools, and resources) are one strategy to disseminate information and seek two-way communication with RE-AIM users. Researchers and practitioners are encouraged to contact the Workgroup regarding resources and needs through the Website (http://www.re-aim.org/contact/).

The Workgroup has focused on making RE-AIM accessible and adaptable across contexts including clinical, community, and other pragmatic settings (28). To accomplish this flexibility, the Workgroup is open to feedback through a variety of formats including contact tools on the RE-AIM website, active presence at conferences, webinars, and presentations.

To support the new research and practice directions summarized above (e.g., enhanced evaluation of contextual factors impacting RE-AIM dimensions; increased application of mixed-methods approaches; more rapid and iterative use of RE-AIM; and combining RE-AIM with other relevant D&I and pragmatic frameworks) we anticipate providing additional resources, application guides, new website features, and more concrete examples of new uses of RE-AIM.

We welcome reader input on these directions and resulting new resource needs. Designing features based on user feedback should enhance the usefulness of the framework and its website (48, 49).

Conclusions

This article summarizes progress by the RE-AIM Workgroup and the open-access resources that the workgroup has developed. The use of RE-AIM has moved beyond its original intent—to improve how programs are evaluated—to being a cornerstone for how programs and research are planned and evaluated, including the implementation phase. The continued popularity of RE-AIM can be attributed to the applicability of the core tenets of the RE-AIM framework across population and settings, its relative ease of use, and understandability to stakeholders (30). We do not see RE-AIM as static, but anticipate that the framework will continue to evolve based on advances in D&I science, to meet user needs and address new applications. For the last 10 years, Workgroup members have used limited resources to ensure that the RE-AIM framework is disseminated to researchers and practitioners. That said, there are ongoing opportunities to enhance the resources, tools and guides provided; therefore, through this paper and our mission, Workgroup members encourage users to take a proactive part in RE-AIM's continual evolution.

Author Contributions

All authors contributed to the conceptualization of the manuscript and its content. All authors contributed to the full manuscript as well as reviewed and approved the final version of the manuscript.

Funding

RG's effort was partially supported by internal funds from the University of Colorado ACCORDS Dissemination and Implementation Science program. PE's time was partially supported by the National Institutes of Health Great Plains IDeA CTR U54GM115458. BR's effort was partially supported by the VA San Diego Center of Excellence in Stress and Mental Health.

Disclaimer

The statements presented in this publication are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

Conflict of Interest

Co-authors are members of the National Working Group on RE-AIM Planning and Evaluation Framework (www.re-aim.org).

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Estabrooks PA, Brownson RC, Pronk NP. Dissemination and implementation science for public health professionals: an overview and call to action. Prev Chronic Dis. (2018) 15:E162. doi: 10.5888/pcd15.180525

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Tabak RG, Reis RS, Wilson P, Brownson RC. Dissemination of health-related research among scientists in three countries: access to resources and current practices. BioMed Res Int. (2015) 2015:179156. doi: 10.1155/2015/179156

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. (2012) 43:337–50. doi: 10.1016/j.amepre.2012.05.024

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav Off Publ Soc Public Health Educ. (2013) 40:257–65. doi: 10.1177/1090198113486805

PubMed Abstract | CrossRef Full Text | Google Scholar

5. EPIS Framework. EPIS Framework. Available online at: https://episframework.com/ (accessed July 5, 2019).

Google Scholar

6. The Consolidated Framework for Implementation Research. The Consolidated Framework for Implementation Research. Available online at: https://cfirguide.org/ (accessed July 5, 2019).

Google Scholar

7. Ford BS, Rabin B, Morrato EH, Glasgow RE. Online resources for dissemination and implementation science: meeting demand and lessons learned. J Clin Transl Sci. (2018) 2:259–66. doi: 10.1017/cts.2018.337

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Dzewaltowski DA, Glasgow RE, Klesges LM, Estabrooks PA, Brock E. RE-AIM: evidence-based standards and a Web resource to improve translation of research into practice. Ann Behav Med Publ Soc Behav Med. (2004) 28:75–80. doi: 10.1207/s15324796abm2802_1

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Glasgow R, Harden S, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a twenty-year review. Front Public Health. (2019) 7:64. doi: 10.3389/fpubh.2019.00064

CrossRef Full Text | Google Scholar

10. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. (2013) 103:e38–e46. doi: 10.2105/AJPH.2013.301299

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to “employ” the RE-AIM model? Eval Health Prof . (2013) 36:44–66. doi: 10.1177/0163278712446066

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/AJPH.89.9.1322

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. (2018) 15:E02. doi: 10.5888/pcd15.170271

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Glasgow RE. RE-AIMing research for application: ways to improve evidence for family medicine. J Am Board Fam Med. (2006) 19:11–9. doi: 10.3122/jabfm.19.1.11

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Manteiga AM, Eyler AA, Valko C, Brownson RC, Evenson KR, Schmid T. The impact of the physical activity policy research network. Am J Prev Med. (2017) 52:S224–7. doi: 10.1016/j.amepre.2016.10.018

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Franz N, Stovall C, Owen M. The perceived value of an extension leadership network: enhancing personal and organizational effectiveness. J Agric Educ Ext. (2010) 16:433–43. doi: 10.1080/1389224X.2010.515066

CrossRef Full Text | Google Scholar

17. Grills NJ, Robinson P, Phillip M. Networking between community health programs: a case study outlining the effectiveness, barriers and enablers. BMC Health Serv Res. (2012) 12:206. doi: 10.1186/1472-6963-12-206

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Brownson RC, Eyler AA, Harris JK, Moore JB, Tabak RG. Getting the word out: new approaches for disseminating public health science. J Public Health Manag Pract. (2018) 24:102–11. doi: 10.1097/PHH.0000000000000673

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. (2015) 350:h2147. doi: 10.1136/bmj.h2147

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. (2009) 62:464–75. doi: 10.1016/j.jclinepi.2008.12.011

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Burke SM, Tomasone JR, Scime NV, Ma JK, Harden SM, Wolfe DL. Physical activity self-management interventions for adults with spinal cord injury: Part 2 – Exploring the generalizability of findings from research to practice. Psychol Sport Exerc. (2018) 37:286–95. doi: 10.1016/j.psychsport.2018.02.013

CrossRef Full Text | Google Scholar

22. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf . (2008) 34:228–43. doi: 10.1016/S1553-7250(08)34030-6

PubMed Abstract | CrossRef Full Text | Google Scholar

23. McCreight M, Rabin B, Glasgow R, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. (2019) 9:1002–11. doi: 10.1093/tbm/ibz085

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Research-Tested Intervention Programs: Home. Available online at: https://rtips.cancer.gov/rtips/index.do. (accessed March 30, 2018).

Google Scholar

25. Boutron I, Altman DG, Moher D, Schulz KF, Ravaud P, CONSORT NPT Group. CONSORT Statement for randomized trials of nonpharmacologic treatments: a 2017 update and a CONSORT extension for nonpharmacologic trial abstracts. Ann Intern Med. (2017) 167:40–7. doi: 10.7326/M17-0046

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. (2016) 2:64. doi: 10.1186/s40814-016-0105-8

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Pinnock H, Barwick M, Carpenter CR, Grandes G, Griffiths CJ, Meissner P, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. (2017) 356:i6795. doi: 10.1136/bmj.i6795

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Harden SM, Smith ML, Ory MG, Smith-Ray RL, Estabrooks P, Glasgow R. RE-AIM in clinical, community, and corporate settings: perspectives, strategies, and recommendations to enhance public health impact. Front Public Health. (2018) 6:71. doi: 10.3389/fpubh.2018.00071

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Belza B, Toobert D, Glasgow R. RE-AIM for Program Planning: Overview and Applications. Available online at: https://fromhungertohealth.files.wordpress.com/2013/02/re-aim_issue_brief.pdf (accessed March 11, 2019).

Google Scholar

30. Ory MG, Altpeter M, Belza B, Helduser J, Zhang C, Smith ML. Perceived utility of the RE-AIM framework for health promotion/disease prevention initiatives for older adults: a case study from the U.S. evidence-based disease prevention initiative. Front Public Health. (2014) 2:143. doi: 10.3389/fpubh.2014.00143

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB, Brito F, et al. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev. (2015) 4:155. doi: 10.1186/s13643-015-0141-0

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns. (2001) 44:119–27. doi: 10.1016/S0738-3991(00)00186-5

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Harden S, Ramalingam N, Breig S, Estabrooks P. Walk this way: our perspective on challenges and opportunities for extension statewide walking promotion programs. J Nutr Educ Behav. (2019) 51:636–43. doi: 10.1016/j.jneb.2018.12.010

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Helmick MJ, Yaroch AL, Parks CA, Estabrooks PA, Hill JL. Utilizing the RE-AIM framework to understand adoption of nutrition policies at food pantries across the USA. Transl Behav Med. (2019) 9:1112–21. doi: 10.1093/tbm/ibz036

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Kennedy S, Smith J, Estabrooks P, McKay H, Noetel M, Nathan N, et al. Scale-up and dissemination of a school-based resistance training program: RE-AIM evaluation of impact. J Sci Med Sport. (2019) 22:S22. doi: 10.1016/j.jsams.2019.08.174

CrossRef Full Text | Google Scholar

36. Shaw RB, Sweet SN, McBride CB, Adair WK, Martin Ginis KA. Operationalizing the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework to evaluate the collective impact of autonomous community programs that promote health and well-being. BMC Public Health. (2019) 19:803. doi: 10.1186/s12889-019-7131-4

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Estabrooks PA, Smith-Ray RL, Dzewaltowski DA, Dowdy D, Lattimore D, Rheaume C, et al. Sustainability of evidence-based community-based physical activity programs for older adults: lessons from Active for Life. Transl Behav Med. (2011) 1:208–15. doi: 10.1007/s13142-011-0039-x

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Yank V, Stafford RS, Rosas LG, Ma J. Baseline reach and adoption characteristics in a randomized controlled trial of two weight loss interventions translated into primary care: a structured report of real-world applicability. Contemp Clin Trials. (2013) 34:126–35. doi: 10.1016/j.cct.2012.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Zoellner JM, You W, Hill JL, Brock DP, Yuhas M, Alexander RC, et al. A comparative effectiveness trial of two family-based childhood obesity treatment programs in a medically underserved region: rationale, design & methods. Contemp Clin Trials. (2019) 84:105801. doi: 10.1016/j.cct.2019.06.015

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Scheirer MA, Santos SLZ, Tagai EK, Bowie J, Slade J, Carter R, et al. Dimensions of sustainability for a health communication intervention in African American churches: a multi-methods study. Implement Sci. (2017) 12:43. doi: 10.1186/s13012-017-0576-x

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Lai J, Klag M, Shikako-Thomas K. Designing a program evaluation for a medical-dental service for adults with autism and intellectual disabilities using the RE-AIM framework. Learn Health Syst. (2019) 3:e10192. doi: 10.1002/lrh2.10192

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Pawlowski CS, Schmidt T, Nielsen JV, Troelsen J, Schipperijn J. Will the children use it?—A RE-AIM evaluation of a local public open space intervention involving children from a deprived neighbourhood. Eval Program Plann. (2019) 77:101706. doi: 10.1016/j.evalprogplan.2019.101706

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Economos CD, Anzman-Frasca S, Koomas AH, Bakun PJ, Brown CM, Brown D, et al. Dissemination of healthy kids out of school principles for obesity prevention: a RE-AIM analysis. Prev Med. (2019) 119:37–43. doi: 10.1016/j.ypmed.2018.12.007

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Estabrooks PA, Gyurcsik NC. Evaluating the impact of behavioral interventions that target physical activity: Issues of generalizability and public health. Psychol Sport Exerc. (2003) 4:41–55. doi: 10.1016/S1469-0292(02)00016-X

CrossRef Full Text | Google Scholar

45. Dzewaltowski DA, Estabrooks PA, Glasgow RE. The future of physical activity behavior change research: what is needed to improve translation of research into health promotion practice? Exerc Sport Sci Rev. (2004) 32:57–63. doi: 10.1097/00003677-200404000-00004

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. (2006) 21:688–94. doi: 10.1093/her/cyl081

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Shoup JA, Gaglio B, Varda D, Glasgow RE. Network analysis of RE-AIM framework: chronology of the field and the connectivity of its contributors. Transl Behav Med. (2015) 5:216–32. doi: 10.1007/s13142-014-0300-1

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Hustak T, Krejcar O. Principles of usability in human-computer interaction. In: Park JJ, Jong H, Chao H-C, Arabnia H, Yen NY, editors. Advanced Multimedia and Ubiquitous Engineering. Lecture Notes in Electrical Engineering. Berlin; Heidelberg: Springer (2016). p. 51–7. doi: 10.1007/978-3-662-47895-0_7

CrossRef Full Text | Google Scholar

49. Garett R, Chiu J, Zhang L, Young SD. A literature review: website design and user engagement. Online J Commun Media Technol. (2016) 6:1–14. doi: 10.29333/ojcmt/2556

PubMed Abstract | CrossRef Full Text | Google Scholar