A conceptual model for building program sustainability in public health settings: Learning from the implementation of the program sustainability action planning model and training curricula

Background: The emergence of implementation science has driven an increase in research examining the implementation of evidence-based programs and policies. However, there has been less attention through program sustainability. To achieve the full benefit of investment in program development and implementation, there must be an understanding of the factors that relate to sustainability; additionally, there is a need for a robust set of tools and trainings to support strategic long-term program sustainability. This paper presents results of our sustainability training intervention and a new conceptual model of sustainability. The proposed conceptual model builds upon the intervention design, further specifying the implementation strategy, strategy-mechanism linkages, and effect modifiers.

Methods: This research is part of the larger randomized control trial evaluating the effectiveness of the Program Sustainability Action Planning Model and Training Curricula. Specifically, this multimethod study establishes a conceptual model for program sustainability and related capacity-building interventions. The training intervention was delivered through workshops and technical assistance to 11 state tobacco control programs, principally entailing the development and implementation of a sustainability action plan. We utilize descriptive statistics and participant perspectives to evaluate the training intervention and propose an empirically-grounded conceptual model for sustainability capacity-building interventions in public health settings.

Results: Participants found intervention components (workshop, workbook, instructor and resources) to be effective. Overall, participants found the intervention improved their ability to develop sustainability action plans and assess their program and partners. Throughout the study, program managers emphasized the importance of the workshop in providing direction for their sustainability work and the value of robust, ongoing technical assistance. Program managers identified several factors that interfered with intervention reception including staff turnover, competing priorities, partnership challenges, and the COVID-19 pandemic.

Conclusion: The present study documents the development and implementation of a novel Program Sustainability Action Planning Model and Training Curricula, one of the first interventions designed to improve program sustainability. In addition, we present an empirically-grounded conceptual model for program sustainability. Considering the paucity of research in this understudied and undefined topic area, this is an important contribution that can serve as a framework for similar intervention designs and implementation efforts.

Clinical Trail Registration: ClinicalTrails.gov identification number is NCT03598114.

Introduction

The emergence of dissemination and implementation (D&I) science has driven an increase in theoretical and empirical understanding of evidence-based program and policy implementation. However, D&I science has paid less attention through the post-implementation period of sustainability. To achieve the full and continued benefits of significant investment in public health research and program development, there must be an understanding of the factors that relate to program sustainability in the post-implementation period; additionally, there is a need for a robust set of tools and trainings to support strategic long-term program sustainability (1).

Despite a burgeoning implementation science literature, there is still a lack of planning for sustainability of evidence-based programs. In a recent review of dissemination and implementation research studies funded by the National Institutes of Health, Johnson et al., found that only only 67.1% of the studies made references to sustainability and none referred to sustainability planning (2). Similarly, there remains a lack of a formally agreed upon definition or elements of sustainability. In Johnson et al.'s review, researchers who actually referenced sustainability in their study, conceptualized sustainability as the “continued delivery of interventions, programs, or implementation strategies,” (2) but there was no formal consensus on the definition. Shediac-Rizkallah and Bone conceptualize sustainability broadly as “the maintenance of health benefits over time” (3). Scheirer and Dearing focus exclusively on the organizational and programmatic elements: “the continued use of program components and activities for the continued achievement of desirable program and population outcomes” (1). In an attempt to define a more formal definition, Moore et al. (4) abstracted sustainability definitions from 209 articles and mapped constructs from the definitions to create a revised definition of sustainability: “after a defined period of time, a program, clinical intervention, and/or implementation strategies continue to be delivered and/or individual behavior change (i.e., clinician, patient) is maintained; the program and individual behavior change may evolve or adapt while continuing to produce benefits for individuals/systems” (4).

We have adopted a more comprehensive definition that considers program organizational components, the evidence for program effectiveness, as well as the process or system in which a program is implemented over time. Comprehensively defined, sustainability is the existence of structures and processes within an adaptive system that allow a program to effectively implement and maintain evidence-based policies and activities that improve health over time (5). This definition is deliberately broad, and moves beyond the characteristics of the program itself to include organizational and other system characteristics. This sustainability definition contains three key elements. First, sustainability is an ongoing (cyclical) change process that requires action-oriented planning to strengthen system capacity (6, 7). Systems include the program, the auspice organization, the community, and the funder. Second, programs rely on structures and processes that contribute toward adequate system capacity as a necessary condition for program sustainability (8). A sustainable program must be integrated into normal organizational operations (9). The characteristics of these programmatic and organizational structures, processes, and community and funder supports (10) build programmatic capacity for sustainability and institutionalization, over time. Finally, what is to be sustained is an evidence-based innovation which is part of a prevention system. Because the innovation is evidence-based, sustainability is essential in attaining positive health impacts (11).

In addition to consensus on one formal definition of sustainability, there remains a lack on congruence in defining “what” factors contribute toward sustainability and “how” programs can achieve sustainability. In considering the factors that contribute toward sustainability Luke et al., identifies 17 frameworks suggesting a variety of factors (with some similarity) that influence program sustainability (12). In addition, only a few conceptual models focus exclusively on the “how” or the programmatic process for building capacity for sustainability. While these frameworks exist, few are actually utilized in D&I research; few researchers funded by the National Institutes of Health referenced frameworks with sustainability constructs and offered limited information on how they operationalized frameworks (2). The Dynamic Sustainability Framework offered by Chambers et al., considers the context in which an intervention is implemented and operationalized within a system (13). However, it does not offer an implementation strategy or mechanism for which programs should engage to improve sustainability of the intervention. May et al.'s, Normalization Process Theory explains how new ideas, ways of acting, and ways of working become routinely embedded or normalized in practice settings (14). It has been utilized in studying program implementation and sustainability (15) and found useful in identifying processes that are likely to enhance sustainability, but again does not offer a mechanism for which programs should engage to improve sustainability. The Program Sustainability Framework (5), which was utilized for our study, outlines eight domains of sustainability including organizational capacity, funding stability, strategic planning, external environment, partnerships, communication, program adaptation, and program evaluation. These domains have been proven to affect the capacity for sustainability among public health programs (5); however, understanding of how these domains interact to improve program sustainability or how to determine how success in one domain might improve other domains is not yet understood.

This paper outlines the development and assessment of the Program Sustainability Action Planning Model and Training Curricula, an intervention which aimed to build capacity for sustainability in state tobacco control programs (TCPs) (16). Specifically, we developed the Program Sustainability Action Planning Model and Training Curricula as an action-oriented training model (defined by Kolb's experiential learning theory) (17) that addressed the internal and external program-related domains outlined in the Program Sustainability Framework (5) proven to affect the capacity for sustainability among public health programs. For the purposes of this paper, the “intervention” is the Program Sustainability Action Planning Model and Training Curricula. We also provide an assessment of the implementation strategy (i.e., use of experiential learning) of this intervention. Using results from this study, including participant perspectives regarding the implementation strategy including intervention component utility and suitability as well as programmatic outcomes, we propose an empirically-grounded conceptual model for implementing sustainability capacity-building interventions in public health settings. The proposed model builds upon and refines the original intervention model used in the development of the intervention, further specifying the implementation strategy, strategy-mechanism linkages, and effect modifiers (preconditions, mediators, and moderators).

Methods

Study design

The Plans, Actions, and Capacity to Sustain Tobacco Control (PACT) study utilized a multiphase approach to develop and assess the effectiveness of a novel intervention, the Program Sustainability Action Planning Model and Training Curricula, to increase the capacity for sustainability among state level tobacco control programs. In the first phase of the PACT study, the intervention was developed through a rigorous multidisciplinary literature review process and a series of expert consultations. We used SCOPUS, ERIC (ProQuest), PubMed, Education Full Text, and PsychINF databases to conduct a formative literature review to inform the development and evaluation of the training intervention. Specifically, we performed formative literature reviews regarding experiential models of learning (i.e., duration and components) and technical assistance (type and duration) to design the intervention. To design the evaluation of the intervention, we conducted formative review to assess previous metrics of experiential learning and technical assistance effectiveness. We also consulted with our PACT advisory team which was comprised of two academic experts in sustainability, two state tobacco control program directors, and three officials from the Centers for Disease Control and Prevention Office of Smoking or Health to determine the final Program Sustainability Action Planning Model and Training Curricula.

In the second phase of this study, a multiyear randomized control trial was conducted to assess the effectiveness in improving the capacity for sustainability [as defined by organization outcomes and Program Sustainability Assessment Test (PSAT) scores] among 24 State Level Tobacco Control Programs (TCP). Ultimately, 11 intervention and 12 control TCPs participated. The Program Sustainability Action Planning Model and Training Curricula was delivered to 11 TCPs through tailored workshops at baseline and ongoing, robust technical assistance through their participation in the multiyear randomized control trial (2018–2022). This paper presents only the results of the evaluation of the training and technical assistance delivered to the 11 intervention states as these data were used in defining the proposed conceptual model.

The program sustainability action planning model and training curricula

The intervention consisted of a two- day workshop to design a program sustainability action plan, two years of tailored technical assistance for implementing the action plan and sustainability outcome assessment. Participants of the workshops actively engaged in developing state TCP-specific sustainability action plans. Each state action plan outlined one or two domain-focused objectives, matched with time-specific activities to be shared across stakeholders present. One person at each workshop claimed responsibility for overseeing the implementation process. Sustainability plans were designed to be implemented over the course of two years. All Program Sustainability Action Planning Training workshops followed the same structure, but were tailored to each state depending on the Program Sustainability Framework domain chosen for the action plan. The two-day workshop involved the TCP staff and as well as a number of stakeholders (i.e., advocates, coalition members, voluntary organizations, grantees, local level health department staff) actively participating to design a sustainability action plan and develop an implementation strategy. Inclusion of and participation by all stakeholders engaged was an important component of the sustainability action plan development process and ensuring all components of the state TCPs were considered. Table 1 outlines the intervention components of this study.

TABLE 1

Table 1. Components of the sustainability action planning training intervention.

Assessing intervention components

Quantitative data methods

Following receipt of the workshop, TCP staff and stakeholders (n = 106) completed a 29-item survey evaluating intervention components for their utility and suitability. Survey items assessing the workshop, workbook, and the instructors utilized a Likert-type scale with anchors “strongly disagree” (1) and “strongly agree” (5). Survey items were grouped by component, forming evaluation areas: Workshop Evaluation, Workbook and Workshop Usefulness, Workbook Satisfaction, and Instructor Evaluation. Survey items were based on the theory of change that allows for study on how a change (intervention) has influenced the design, implementation, and institutionalization of a program and were designed specifically to assess outcomes related to Kolb experiential learning components (17). Descriptive statistics were calculated at the item- and component-level to analyze evaluation survey data.

Qualitative data methods

Workshop and workbook evaluation

Included in the evaluation survey of the intervention were two open-ended questions, (1) What were the three most important things you learned at this training? Please explain and (2) What suggestions do you have for improving this workshop? A total of 220 answers were recorded in response to most important things learned at the training. Responses such as, None, and Not applicable, were excluded from analysis (n = 13). The remaining 207 responses were reviewed and grouped into themes. A total of 100 answers were recorded in response to suggestions for improvement. Responses such as, None, and Not applicable, were excluded from analysis (n = 37). The remaining 63 responses were reviewed and grouped into themes.

Technical assistance (TA) calls

From December 2018 to January 2022, staff from the 11 intervention TCPs participated in TA calls to assess their progress on implementing their sustainability plans and to determine challenges and resource needs. Each state received an average of 2 calls/year during their two years of study participation (n = 46). These calls lasted about 30–45 min and were audio recorded.

Each TA call record was professionally transcribed using an online service (Rev.com). Transcripts were reviewed for accuracy and uploaded to NVivo 20 (released in March 2020) for coding. We used an inductive approach for thematic analysis. We developed a codebook based on the items addressed in the TA calls. The codes and sub-codes of the initial codebook were revised throughout the coding of transcripts. The final codebook consisted of four codes and sixteen sub-codes.

For the coding process, three research team members coded transcripts until they reached substantial inter-rater reliability (kappa = .72) (18). Differences between coders were discussed and addressed. Remaining transcripts were coded by a single research team member. This work focuses on the themes from two parent codes: (1) developing capacity for sustainability and (2) overall study feedback.

Diagram modelling

Quantitative and qualitative intervention component data were fit to the original intervention model through an Agile Science-informed causal pathway diagram modeling process to propose a generalizable, empirically-grounded conceptual model for implementing sustainability capacity-building interventions in public health settings. An Agile Science informed process was utilized due to the focus on intervention modularity and condition specification. As outlined by Lewis et al. (19), in implementation research this process entails specifying implementation strategies, strategy-mechanism linkages, effect modifiers, and distal and proximal outcomes (19). In the present study, the research team carried out the diagram modeling process over multiple working sessions. The research team presented results to the PACT advisory team, to ensure face and content validity.

Results

Assessing intervention components

Quantitative results

Workshop evaluation

The workshop evaluation area comprised nine survey items assessing logistics, utility, suitability, and outcomes. At the component level, the workshop was favorably assessed by intervention participants (M = 4.24, SD = 0.81). At the item level, participants indicated that the workshop augmented their understanding of the action planning process and their capacity to move these plans forward (M = 4.34, SD = 0.69; M = 4.37, SD = 0.65), and was overall beneficial for their program (M = 4.35, SD = 0.69). Participants indicated that they planned to translate workshop learning objectives into their tobacco control work (M = 4.47, SD = 0.59). See Table 2.

TABLE 2

Table 2. Program sustainability action planning model and training curricula evaluation.

Workbook and workshop usefulness

The workbook and workshop usefulness area comprised 6 survey items assessing the usefulness of intervention component modules: Defining Program (M = 4.21, SD = 0.74), Understanding Sustainability (M = 4.30, SD = 0.61), Reflecting on Results (M = 4.26, SD = 0.70), Building an Action Plan (M = 4.49, SD = 0.61), Continuing Progress (M = 4.14, SD = 0.69), and Appendix Resources (M = 3.89, SD = 0.83). At the component level, participant responses indicate that the workbook and workshop were useful (M = 4.22, SD = 0.72) (see Table 2).

Workbook satisfaction

The workbook satisfaction area comprised 4 survey items assessing design (M = 4.48, SD = 0.65), content organization (M = 4.50, SD = 0.59), ease of use (M = 4.55, SD = 0.63), and instructional clarity (M = 4.41, SD = 0.65). At the component level, participants were highly satisfied with the workbook (M = 4.48, SD = 0.63) (see Table 2).

Instructor evaluation

To evaluate the workshop training instructor, we used seven survey items assessing their subject matter expertise and professionalism. At the item level, participants reported that workshop instructors demonstrated a thorough knowledge of the subject matter (M = 4.68, SD = 0.56), used effective teaching and facilitating techniques (M = 4.58, SD = 0.66), and discussed how workshop learning objectives could be applied to their tobacco control work (M = 4.57, SD = 0.62). At the component level, instructors were favorably assessed (M = 4.70, SD = 0.53) (see Table 2).

Thematic analysis results

Workshop and workbook evaluation

Overall, participants found that the sustainability training intervention provided them with a better understanding of program sustainability and improved their ability to develop sustainability action plans. Participants noted it was useful to have a common language around sustainability and a shared action plan to be completed collaboratively by the TCP and its partners.

It was very important and helpful to have common language and definitions for the domains –State Health Department staff

Participants also shared that the workshop enhanced their ability to assess the strengths and weaknesses of their program. The workshop also helped participants gain a better understanding of their partners, noting how valuable it was to have dedicated time with their partners and include them in their sustainability action plan.

…just working through the actual action plan because we're all so busy and so being able to do that and get that started in that meeting. I don't how long it would have taken us to get that done otherwise. –State Health Department, Director

Program managers emphasized the importance of the workshop in providing direction for their sustainability work throughout the study.

Technical assistance

The TCP managers often commented on the importance of robust, ongoing technical assistance to continue implementing their sustainability plan. Ongoing access to program sustainability resources and intervention-facilitated peer learning were frequently requested during TA calls to further implementation of their action plans.

I think it's always been nice to have timely reminders to focus on the strategic planning process and make sure that we're continually looking at how we're tracking against those milestones, the way outlined in the sustainability plan. That's been really useful for me … –State Health Department, Program Manager

Managers also felt that the alignment of their action plan with other grant requirements facilitated implementation. Strong partner engagement was also identified by program managers as a factor which augmented sustainability training and technical assistance effectiveness.

I appreciate the technical assistance and all of the tools that you put together for us to use. It was really, really helpful and it helped catalyze some really robust conversations at some of the stakeholder meetings we've had …  –State Health Department, Program Manager

Program managers identified several factors that interfered with their progress in implementing their plans throughout the study. Most notable were high levels of staff turnover, competing priorities, partnership challenges, and the COVID-19 pandemic. In discussing staff turnover, program managers shared they were responsible for covering the duties of the vacant positions in their TCP making it difficult to prioritize their action plan. They also noted the vacancies they experienced in their program were at times the individuals directly responsible for aspects of the action plan, which slowed down and at times halted their progress.

While that position is vacant, I'm kind of doing double duty, which just makes it hard to be focused on longterm just because of dealing with the day to day. We still try to do that as much as we possibly can, but I'm definitely looking forward to getting that position filled so we can share that workload a little bit. –State Health Department, Policy Section Manager

Regarding competing priorities, managers discussed the prioritization of funding and contractual agreements, as well as emergency bans on tobacco products which required immediate attention.

Plus we had that emergency ban on vapor products, flavored vapor products that kind of inundated our time. So, we had to push a lot of our dates back, and so that's what we did. So, we spent a lot of time revising the action plan in terms of dates and really moving towards completing some of the activities that we had said that we would do by January. –State Health Department, Unit Manager

Some program managers noted they struggled to maintain engagement with their partners throughout their participation in the study, leading to slower responses to communication and lack of follow through on action plan responsibilities.

Trying to make sure that partners were continually engaged and that when they left the meeting, that they felt empowered enough to actually follow through on activities that have been discussed. Because sometimes we would have meetings and everybody would leave with what they were supposed to do but when we met again there hadn't been any movement on anything. –State Health Department, Program Director

Finally, the COVID-19 pandemic impacted the overall capacity of TCPs, as public health staff were reassigned to work on pandemic response activities.

And we also were impacted with COVID and capacity, so we definitely had a plan and we had a product to share with legislators, but one of our staff has been and remains full-time on COVID … –State Health Department, Program Manager

…I'm sure you've heard this from other people, but partners are dealing with COVID too… they don't have the time or ability to sit in on some of the meetings that they normally would with us. So that's been somewhat of an issue as well. –State Health Department, Program Director

When asked about recommendations for sustainability training and technical assistance intervention improvement, participants wanted it to be more adaptable to the changing needs and priorities of their TCP. They also requested additional opportunities to interact with other states enrolled in the study to further peer learning.

I think, as we go through the strategic planning process, if you have seen strategic plans from other States that you think look amazing, feel free to send them… if you've been working with other States and you're really impressed with the caliber of work that they're doing, they have innovative ideas, I definitely think it's a good idea to learn from the best so that we might be able to emulate. –State Health Department, Program Manager

The conceptual model

Using quantitative and qualitative data we engaged in an Agile Science-informed causal pathway diagram modeling process (19) to develop our conceptual model (Figure 1). The implementation strategy consisted of the Program Sustainability Action Planning Model and Training Curricula. The mechanism through which the Program Sustainability Action Planning Model and Training Curricula intervention was implemented was through action oriented, participatory training, as defined by Kolb's experiential learning theory (17). Both quantitative and qualitative results indicate the Program Sustainability Action Planning Model and Training Curricula was an effective intervention in developing and implementing a sustainability action plan.

FIGURE 1

Figure 1. Conceptual model for building program sustainability in public health settings.

Moderating factors or factors that increase or decrease the level of influence of the implementation of the state TCP sustainability action plans included high levels of staff turnover, competing priorities, partnership challenges, and the COVID-19 pandemic. While the pandemic was specific to this project's timing, other major public health events could be considered.

Mediating factors or variables that influence the outcome of the implementation strategy included programmatic and organizational factors, community partner support and funder support. Programmatic factors contributing toward the success of sustainability action plan implementation included dedicated time to work on sustainability plan activities and leadership support in continuing to work on the plans in spite of other competing priorities. Those state TCPs experiencing staff vacancies, especially involving those staff responsible for implementing the plan, slowed the ability to complete plan activities. Qualitative results indicated TCPs with strong partner engagement throughout their implementation process were more successful in completing their objectives and goals outlined in their sustainability plans. Likewise, some programs were unable to complete certain objectives outlined because they struggled to maintain engagement of their partners throughout their participation in the study. Competing priorities or lack of budgetary funding support for TCPs deterred progress on TCP sustainability action plan implementation (see Figure 1).

Discussion

The present study makes several important contributions to implementation science. Notably, the present study documents the development and implementation of the novel Program Sustainability Action Planning Model and Training Curricula, one of the first proposed training interventions for improving the capacity for program sustainability in public health. We also propose an empirically-grounded conceptual model for implementing sustainability capacity-building interventions in public health settings. The proposed model builds upon previous work and specifies the implementation strategy, strategy-mechanism linkages, and effect modifiers (preconditions, mediators, and moderators).

Our results indicate that Program Sustainability Action Planning Model and Training Curricula was an effective capacity building intervention in developing and implementing a sustainability action plan. Those in receipt of the Program Sustainability Action Planning Model and Training Curricula assess it favorably in regards to its utility and suitability, across all evaluation areas. Thematic analysis further qualified quantitative results: participants indicated that the Program Sustainability Action Planning Model and Training Curricula enhanced their understanding of program sustainability and related program-specific characteristics as well as the role of TCP partners. In addition, throughout the study, program managers emphasized the importance of the workshop in providing direction for their sustainability work and the value of robust, ongoing technical assistance. Ongoing access to program sustainability resources and intervention-facilitated peer learning and partner engagement were also noted as factors augmenting Program Sustainability Action Planning Model and Training Curricula intervention component effectiveness.

We used both qualitative and quantitative results in refining a conceptual model for implementing sustainability capacity-building interventions in public health settings. Other studies focused on understanding sustainability have recognized similar components (mediators and moderators) identified in our proposed model but none have considered the complete process including the implementation strategy and mechanism for which to plan for and improve program sustainability. For instance, when utilizing the Normalization Process Theory in evaluating the implementation of an evidence-based violence screening model, Hooker et al., found several organizational components that mediated program “normalization” or programmatic sustainability including lack of staff capacity (15). Similar to our results, authors also noted the importance of partner and community interaction (collective action) in achieving desired results. Finally, authors also noted the importance of tracking (reflective monitoring) results. This is similar to the idea of the inclusion of active tracking of sustainability action plans in this study. Though not broadly applied, The Dynamic Sustainability Framework also defines similar components to our conceptual model. Specifically, the idea that systems are not static, but rather dynamic forcing programs to be adaptive and ready to respond to a constantly changing environment to be sustainable (13). In addition to the advent of new programmatic components or evidence invoking the need for change of a program, there exist external factors and events that require a dynamic response by programs. COVID-19 and changes in tobacco regulatory requirements were found to influence sustainability in this study.

Limitations

There are several limitations to this study. First, although evidence-informed and developed through a systematic process, the proposed model has not been empirically tested. The present study also utilized programmatic data and the perspectives of individuals at 11 state TCPs. Although this is of sufficient size to statistically evaluate the intervention for its effectiveness, findings may not be completely generalizable across all state TCPs. Further work is therefore needed to establish generalizability. Finally, our data were generated throughout the COVID-19 pandemic, and therefore are not unaffected by the unprecedented conditions the event produced. Throughout this time state TCPs operated in nonroutine ways, and the extent to which emergent phenomena in our study were products of this is unknown. While we feel that any public health event might disrupt the state health department system, one may not have the same magnitude effect.

Conclusion

By establishing a method for action planning and technical assistance for program sustainability, the present study supports public health programs broadly in their understanding of and achievement in sustainability, an outcome that has become increasingly critical given the environmental complexity. In addition, this work advances the field of study regarding action planning and technical assistance, which contributes to implementation science beyond the topic area of sustainability. Finally, the present study advances implementation science by establishing an empirically-grounded conceptual model for program sustainability and related capacity-building interventions. Considering the paucity of research in this understudied and undefined topic area, this is a significant contribution that can serve as a framework for similar intervention designs and implementation efforts. Future research in the application of this framework will be beneficial in defining its utility and refining its components.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Human Research Protection Office, Washington University. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

SMR guided study design and led and contributed to writing the manuscript. JG managed the development of the interview guide, conducted interviews, coded interview transcripts, analyzed and interpreted interview themes, and was a contributor in writing the manuscript. EJ analyzed quantitative data and was a contributor in writing the manuscript. All authors contributed to the article and approved the submitted version.

Funding

The present study is funded by the National Cancer Institute of the National Institutes of Health under award number R01CA203844. The findings and conclusions of this study are those of the authors and do not necessarily represent the official positions of the National Institutes of Health.

Acknowledgments

We would like to acknowledge Louise Farah Saliba for her work in managing the PACT study. Her work and dedication allowed us to complete the project successfully. We would also like to acknowledge the rest of the PACT study team, Ross Brownson, Douglas Luke, Todd Combs, Kim Prewitt, and Rachel Hackett for their input and work related to completing PACT study activities.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. (2011) 101:2059–67. doi: 10.2105/AJPH.2011.300193

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Johnson AM, Moore JE, Chambers DA, Rup J, Dinyarian C, Straus SE. How do researchers conceptualize and plan for the sustainability of their NIH R01 implementation projects? Implement Sci. (2019) 14:50. doi: 10.1186/s13012-019-0895-1

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. (1998) 13(l):87–108. doi: 10.1093/her/13.1.87

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Moore JE, Mascarenhas A, Bain J, Straus SE. Developing a comprehensive definition of sustainability. Implement Sci. (2017) 12:110. doi: 10.1186/s13012-017-0637-1

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Schell SF, Luke DA, Schooley MW, Elliot MB, Herbers SH, Mueller N, et al. Public health program capacity for sustainability: a new framework. Implement Sci. (2013) 8:15. doi: 10.1186/1748-5908-8-15

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Osch LV, Lechner L, Reubsaet A, Vries HD. From theory to practice: an explorative study into the instrumentality and specificity of implementation intentions. Psychol Health. (2010) 25(3):351–64. doi: 10.1080/08870440802642155

PubMed Abstract | CrossRef Full Text | Google Scholar

7. de Vries H, Eggers SM, Bolman C. The role of action planning and plan enactment for smoking cessation. BMC Public Health. (2013) 13:393. doi: 10.1186/1471-2458-13-393

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Friedman L, Schreiber L. Healthy families America state systems development: an emerging practice to ensure program growth and sustainability. J Prev Interv Community. (2007) 34(1–2):67–87. doi: 10.1300/J005v34n01_04

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Edwards NC, Roelofs SM. Sustainability: the elusive dimension of international health projects. Can J Public Health. (2006) 97(1):45–9. doi: 10.1007/BF03405214

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Savaya R, Spiro S. Predictors of sustainability of social programs. Am J Eval. (2012) 33(1):26–43. doi: 10.1177/1098214011408066

CrossRef Full Text | Google Scholar

11. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. (2008) 34(4):228–43. doi: 10.1016/S1553-7250(08)34030-6

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The program sustainability assessment tool: a new instrument for public health programs. Prev Chronic Dis. (2014) 11:130–84. doi: 10.5888/pcd11.130184

CrossRef Full Text | Google Scholar

13. Chambers Da, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. (2013) 8(1):117. doi: 10.1186/1748-5908-8-117

PubMed Abstract | CrossRef Full Text | Google Scholar

14. May CR, Mair F, Finch T, MacFarlane A, Dowrick C, Treweek S, et al. Development of a theory of implementation and integration: normalization process theory. Implement Sci. (2009) 4(1):1–9. doi: 10.1186/1748-5908-4-1

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Hooker L, Small R, Humphreys C, Hegarty K, Taft A. Applying normalization process theory to understand implementation of a family violence screening and care model in maternal and child health nursing practice: a mixed method process evaluation of a randomised controlled trial. Implement Sci. (2015) 10(1):1–13. doi: 10.1186/s13012-015-0230-4

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Vitale R, Blaine T, Zofkie E, Moreland-Russell S, Combs T, Brownson RC, et al. Developing an evidence-based program sustainability training curriculum: a group randomized, multi-phase approach. Implement Sci. (2018) 13:1. doi: 10.1186/s13012-018-0819-5

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kolb DA. Experiential learning: Experience as the source of learning and development. 1st ed. Englewood Cliffs, NJ: Prentice Hall, Inc. (1984). doi: 10.1002/job.4030080408

18. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. (1960) 20(1):37–46.

19. Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, Walsh-Bailey C, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. (2018) 6:136. doi: 10.3389/fpubh.2018.00136

PubMed Abstract | CrossRef Full Text | Google Scholar