Co-creating physical activity interventions: Findings from a multiple case study using mixed methods

Introduction: In health and physical activity promotion, there is growing interest in co-creation approaches that involve researchers and non-academic stakeholders in developing new interventions. Previous research has shown the promising results of cooperative planning as a co-creation approach in building new capacities and implementing physical activity-promoting interventions in nursing care and automotive mechatronics. However, it remains unclear whether (1) cooperative planning for physical activity promotion can be successfully transferred to other settings in the nursing care and automotive mechatronic sectors and (2) what key factors influence its success or failure.

Methods: We conducted a multiple case study in three settings in the nursing care and automotive mechatronics sectors. Following a mixed methods approach, we collected, analyzed, and triangulated data from documents (n = 17), questionnaires (n = 66), and interviews (n = 6). Quantitative data were analyzed descriptively and through using nonparametric analyses of variance; qualitative data were analyzed using qualitative content analysis by extraction.

Results: The transfer of cooperative planning to new settings was realized, though the impact varied by setting. While the interventions were developed and implemented in nursing care settings, interventions were developed but not implemented in the automotive mechatronics setting. In this context, intervention implementation was influenced by 11 key factors: champion, commitment, embedment, empowerment, engagement, health-promoting leadership, ownership, relevance, resources, responsibility, and strategic planning. Furthermore, the transfer of cooperative planning was influenced by different activity characteristics, namely elaboration & reconsideration, group composition, number of meetings, participation, period, prioritization, and researchers' input & support.

Discussion: The present article contributes to a better understanding of a co-creation approach utilized for physical activity promotion and provides new insights into (1) the transferability of cooperative planning and (2) the associated key factors influencing intervention implementation. The success of cooperative planning varied by setting and was influenced by several activity characteristics and key factors, some of which showed complex relationships. This raises the question of whether some settings might benefit more from a co-creation approach than others. Therefore, future co-creation initiatives should carefully consider the specific characteristics of a setting to select and apply the most appropriate approach.

Introduction

Synergizing the scientific world with the real world is considered a key benefit of co-creation (1). Indeed, co-creation approaches, in which researchers collaborate with non-academic stakeholders (i.e., end-users, practitioners, policy-makers) (2), are increasingly used to develop health-promoting interventions tailored to end-users and the given setting. On the one hand, tailoring interventions to end-users can increase their acceptability (3, 4) and effectiveness (5, 6). On the other hand, adapting interventions to the setting facilitates its contextualization by embedding these interventions into established routines and structures, utilizing existing resources, and building new capacities (7–9), in turn increasing the likelihood of sustained implementation (10–13).

In particular, the postulated fit of co-created interventions through the development of solutions that are suited to local circumstances makes this approach an appealing one for population groups that are characterized by specific needs and resources. Employees with higher levels of occupational physical activity (PA) are one such population group because PA is associated with fewer beneficial health effects for this group compared with employees with lower levels of occupational PA (14, 15). Following this, fostering the competencies needed to master physical demands in a healthy manner and adopt a physically active lifestyle might be a good focus of PA promotion for people with physically demanding occupations, rather than focusing solely on increasing PA levels (16).

Against this background, the research project Physical Activity-related Health Competence in Apprenticeship and Vocational Education (PArC-AVE), which was embedded in the research consortium Capital4Health, focused on PA promotion in the automotive mechatronics and nursing care sectors using a co-creation approach called cooperative planning (CP) (17, 18). CP engages non-academic stakeholders, including members of the target population, and researchers in an equal decision-making process to plan, develop, and implement interventions (19). Thus, CP exhibits parallels with other participatory or co-creation approaches (e.g., intervention mapping or community-based participatory research), but offers a unique constellation by combining the four key components of theory and goal orientation, involvement of all relevant stakeholders, knowledge co-production, and the use of progress monitoring and feedback loops (20). In the PArC-AVE project, the primary aim was to develop and implement new interventions to facilitate PA promotion within the given setting while taking the needs and resources of the end-users and setting into account. During the participatory development and implementation of the interventions involving end-users and other relevant actors from research, policy, and practice, the focus was on both the structural level by creating a PA-friendly environment and the individual level by promoting end-users' PA and physical activity-related health competence (PAHCO) (21, 22). Previous research examining CP in nursing care and automotive mechatronics has shown promising results when it comes to building new capacities and (sustainably) implementing PA-promoting interventions (23, 24).

Taking into account the concept of scaling up (25, 26) raises the question of the transferability of such approaches or interventions, i.e., the extent to which their impact could be achieved in another setting (27). More precisely, in our case, it remains unclear whether CP for PA promotion can be successfully transferred to other settings in the nursing care and automotive mechatronic sectors to reach and benefit more employees with physically demanding occupations. Additionally, the question arises as to what factors influence the success or failure of CP as a co-creation approach for PA promotion. The increasing number of studies using CP (28–31) or similar strategies in PA promotion and health promotion (11, 32, 33), along with the critical voices discussing the limitations and challenges of co-creation, such as the resources required or the risk of conflicts because of different interests (34, 35), underscore the need to explore these unanswered questions. Thus, the current study aims to address the following research questions:

1. How (un)successful is the transfer of CP for PA promotion to other settings in the nursing care and automotive mechatronic sectors? (transferability).

2. What key factors influence the success or failure of CP for PA promotion and, in particular, intervention implementation? (key factors).

Methods

Overview

To answer both research questions, we have used a multiple case study design with three different settings in the nursing care and automotive mechatronic sectors, with each representing one case. In each setting, a separate CP process was initiated in 2018 to develop and implement new multi-component interventions, each comprising multiple PA-promoting intervention components. These intervention components were expected to work best when implemented in combination but could also be implemented separately. The intervention implementation was not limited in time but was instead intended for the long term, if possible. The program activities, underlying evaluation theory, and planned methods have already been reported in detail in a study protocol (36). In brief, the evaluation of the transferability of CP and the key factors influencing its success or failure were based on a logic model illustrating the assumed mode of action of CP (see Figure 1). The logic model component Activities includes all project meetings and visits in the settings. Outputs are the direct products of the CP process, that is, the developed interventions documented in action plans, while Outcomes are the subsequent changes at the structural and individual levels. Contextual factors are defined as those factors influencing the CP process and its success or failure; these consist of factors that have been predefined based on previous project findings and a literature screening, as well as additional factors that have not yet been identified (36). Following the principles of a pragmatic evaluation (37, 38), we used a mixed methods approach to examine (1) the successful transfer of CP based on the Activities, Outputs, and Structural outcomes and (2) the key factors influencing the success or failure of CP, particularly the intervention implementation based on the Contextual factors. By comparing the results of all three settings, similarities and differences could be identified and aggregated to answer both research questions.

FIGURE 1

Figure 1. Logic model of the PArC-AVE project (including research questions). CP, cooperative planning; PA, physical activity; PAHCO, physical-activity related health competence.

Cases and participants

The multiple case study was undertaken in two state vocational education centers for health professions (Setting A: 200 nursing students enrolled in a nursing program, localized in a small city; Setting B: 180 nursing students enrolled in a nursing program, localized in a metropolis), and the assembly department of an automotive manufacturer (Setting C: 12,000 employees in the assembly department, localized in a large city), all located in Germany. The participants included end-users and other stakeholders involved in the CP processes. Table 1 provides more information about the final sample listed by the data sources.

TABLE 1

Table 1. Description of the sample split by data sources.

Data collection

Data were collected using quantitative and qualitative methods. To assess the transferability of CP based on planning meetings (Activities) and resulting multi-component interventions (Outputs, Outcomes), the data from structured minutes (qualitative), action plans (qualitative), questionnaires (quantitative), and interviews (qualitative) were used. Key factors influencing CP, particularly the intervention implementation (Contextual factors), were examined based on data from questionnaires (quantitative) and interviews (qualitative). The time points of measurement for all data sources are presented in Figure 2.

FIGURE 2

Figure 2. Cooperative planning processes and moments of data collections. CP, cooperative planning; ORIC, organizational readiness for implementing change.

Quantitative data

In both questionnaire surveys, we used a maximum variation sampling technique to select the participants (39). To assess organizational readiness for change (40) as a predefined factor influencing CP, all stakeholders who attended the first planning meeting in each setting were invited to complete the Organizational Readiness for Implementing Change (ORIC) questionnaire (41), which had been translated into German [see study protocol (36)] at the beginning of the planning phase in September 2018 and January 2019. The questionnaire consisted of 12 items answered on a 5-point Likert scale (1 = disagree; 5 = agree). In this questionnaire survey, “change” refers to changes at the organizational level targeting PA promotion in the PArC-AVE project.

Furthermore, we utilized setting-specific CP questionnaires to evaluate the organization and realization of planning meetings, implementation status of intervention components, appraisal of the multi-component intervention, and predefined factors influencing CP. The development of these questionnaires is described in the study protocol (36); an overview of all items and subscales can be found in Supplementary material 2. The items were answered on a 5-point Likert scale (1 = disagree; 5 = agree). All stakeholders who attended at least one planning meeting in each setting were invited to complete the CP questionnaire from September to October 2020 in an online format using SoSci Survey ver. 3.2.12 (SoSci Survey GmbH, Munich, Germany).

Qualitative data

To collect detailed information on the planning meetings and number and characteristics of the involved actors, we took structured minutes of all planning meetings from September 2018 to November 2019. At the end of the planning phase, an action plan was created for each setting, documenting the number and description of the multi-component interventions developed (July–November 2019).

To identify the key factors influencing CP, particularly intervention implementation, we conducted semistructured interviews from December 2020 to January 2021. We developed setting-specific interview guides by building on data collected via structured minutes, action plans, and questionnaires (see Supplementary material 1). Following a purposeful sampling strategy of information-rich cases (39), we sought key informants with great knowledge about and influence on the PArC-AVE project. Accordingly, we selected two main stakeholders from each setting who were our contact persons and/or were substantially involved in the development and implementation of the interventions for the interviews. In Setting A, one invited stakeholder declined to participate because of a high workload, so another involved stakeholder representing a similar perspective was asked to participate. Two authors (EG and JP) conducted the interviews using the teleconferencing software Zoom Cloud Meetings (Zoom Video Communications, Inc., San Jose, USA). The interviews were audio-recorded and, on average, lasted about an hour (SD = 26.27; range 35.88–103.23 min).

Data analyses

Quantitative data

Following the psychometric assessment studies by Shea et al. (41), we used the revised 10-item version of the ORIC questionnaire and analyzed mean scores of the 10-item total ORIC scale, the 5-item Change Commitment subscale, and the 5-item Change Efficacy subscale. The non-parametric Kruskal-Wallis test and Dunn-Bonferroni post-hoc tests were employed to examine differences across the settings. To compare the characteristics of the CP processes between the settings (i.e., planning meetings, implementation status, interventions' appraisal, influence of predefined factors), the CP questionnaire data were analyzed using the non-parametric Kruskal-Wallis test. Additionally, semantic differential charts were used to visualize the organization and realization of planning meetings and the influence of predefined factors across settings. The statistical analyses were performed using IBM SPSS Statistics ver. 26 (IBM, Armonk, USA); Microsoft Excel 2016 (Microsoft Corporation, Redmond, USA) with XLSTAT was used for the descriptive analysis. A significance level of p < 0.05 was applied for all analyses.

Qualitative data

The structured minutes and action plans were analyzed regarding the number and characteristics of planning meetings, involved actors, and intervention components using Microsoft Excel 2016 (Microsoft Corporation, Redmond, USA). The interviews were transcribed verbatim. Although analysis of the interview transcripts using qualitative content analysis according to Kuckartz (42) was initially planned in the study protocol (36), we decided to apply the qualitative content analysis procedure according to Gläser and Laudel (43, 44) instead. The main reason for this change was that Gläser and Laudel's content analysis focuses on the reconstruction of causal relationships, that is, between processes and outcomes, which is not supported by the coding procedure according to Kuckartz in this form. According to Gläser and Laudel (43, 44), the analysis starts with a set of theoretically derived categories, which is subsequently used for extracting relevant information from the interview transcripts. In our case, we referred to the logic model and our research questions to deductively define the categories of activity characteristics influencing the transfer of CP and key factors influencing intervention implementation. Then, two authors (EG and JP) developed the extraction rules, extracted the information from the text, and generated two extraction tables, one for each category. These tables include all information from the transcripts assigned to the respective categories. More precisely, the information was extracted in the following format: subject (one characteristic of the respective category labeled with a keyword), content (more detailed description of the subject), reported cause (information about a cause of the subject) and/or reported effect (information about the effect of the subject), and source (link to the relevant text passage in the transcript). These extraction tables were subsequently sorted by setting; the subjects were thematically grouped and summarized where appropriate and subsequently analyzed within and across settings. Microsoft Word 2016 (Microsoft Corporation, Redmond, USA) with MIA software (Ger.: Makrosammlung für qualitative Inhaltsanalyse; Eng.: macro collection for qualitative content analysis) (45) was used for the qualitative data analysis.

Data triangulation

Following the separate analyses, the quantitative and qualitative data were triangulated at the interpretation stage (46, 47) to provide a comprehensive description of transferability and key factors. First, the quantitative and qualitative findings were triangulated separately for each setting by identifying and comparing the main findings. Subsequently, patterns of similarity or difference were examined among the three settings. Two researchers (EG and JP) participated in the triangulation procedure to minimize potential bias in analyzing and interpreting the different findings. For discrepancies between researchers, consensus was reached through discussions.

Results

To present the results split by the research questions, we built on the previously described and assigned logic model components of Activities, Outputs, Structural outcomes, and Contextual factors.

Transferability: Success or failure of the transferred CP processes

Activities

Our analysis of structured minutes revealed differences in the number and time periods of meetings and number of involved actors among the three settings. The number of meetings varied from four in Settings A and B to six in Setting C, with each meeting lasting 3 h. The meetings took place over a period of 10 months in Setting A, 7 months in Setting B, and 14 months in Setting C. The number of involved actors varied between 17 and 19 in Setting A (M = 17.8), 7 and 15 in Setting B (M = 13.0), and 5 and 13 in Setting C (M = 10.5). The involved actors were researchers (Settings A, B, C: professor, research assistants) and non-academic stakeholders such as practitioners (Settings A, B: teachers, head of the nursing education program; Setting C: occupational physicians, occupational health referents, representative of the health insurance company, member of the works council, training center staff, assembly department manager), end-users (Settings A, B: nursing students; Setting C: assembly workers), and policy-makers (Settings A, B: headmasters, head of the nursing school; Setting C: none).

The analysis of the CP questionnaire data on the organization and realization of planning meetings yielded conspicuous findings. Across all settings, we found no significant differences for the items of the subscale research, namely, researchers' input, organization, guidance, and goal setting during CP. For example, the researchers' input revealed no significant differences across the settings (H(2) = 0.56, p = 0.755). However, for the other subscales stakeholders, planning group, and benefits of CP, significant differences between the settings were found for at least one item. For example, in terms of perceived benefits, significant differences across settings were identified for the perceived relevance of PA and health (H(2) = 11.86, p = 0.003), with higher scores for Setting A compared with Setting C. Details of the descriptive analysis and the significant differences for all subscales and across settings are presented in Supplementary material 2.

Qualitative content analysis of the interview data revealed the following seven subjects for the category activity characteristics influencing the transfer of CP: elaboration & reconsideration, group composition, number of meetings, participation, period, prioritization, and researchers' input & support (for detailed information, see Figure 3). In addition to the identified activity characteristics, we found numerous effects of these. For example, in all three settings, the identified activity characteristic researchers' input & support led to a high relevance of the project within the setting. In Settings A and B, the intensive elaboration & reconsideration during intervention development, the participation of relevant stakeholders (i.e., nursing students and teachers), and the period including timing and regularity of meetings had positive effects (e.g., the elaboration & reconsideration and participation positively influenced the empowerment of stakeholders to contribute to the intervention implementation). However, the absence of relevant stakeholders in the planning group (group composition) had negative effects in Settings B and C, such as a missing definition of responsibilities or low engagement of stakeholders to contribute to the intervention implementation. In Setting C, the insufficient number of meetings and prioritization of collected ideas during one meeting also had a negative effect. For example, prioritizing ideas led to a loss of innovation in interventions, in turn reducing commitment to the project and interventions. All discovered causal relationships between the identified activity characteristics and effects for the three settings are visualized in Figure 3.

FIGURE 3

Figure 3. Activity characteristics influencing the transfer of cooperative planning identified through qualitative content analysis.

Outputs and outcomes

According to the action plans, the planning meetings resulted in one newly developed multi-component intervention per setting, including 12 intervention components in Setting A, 11 in Setting B, and six in Setting C. Examples of the single components are the provision of information (intervention component information for teachers in Setting A), competence training (intervention component training module PAHCO in Setting C), or PA programs (intervention component BuG lesson in Settings A and B). An overview of the interventions, including a description of each intervention component, is provided in Supplementary material 3.

The analysis of the CP questionnaire data on the current implementation status and expected sustainability of the individual intervention components revealed heterogeneous results across the settings (see Supplementary material 3). Notably, only a few of the participants had information on the implementation status and sustainability of the intervention components; in addition, the participants' responses were not always consistent. Thus, the data analysis was based on an agreement rate of at least 66.7% (i.e., more than two thirds of the participants with information gave the same response) to make conclusive statements about the implementation status and expected sustainability. Overall, 33.3% of the intervention components (n = 4) were implemented in Setting A and 18.2% (n = 2) were implemented in Setting B. In Setting C, 33.3% of the intervention components (n = 2) were not perpetuated, and 16.7% (n = 1) were not implemented because of COVID-19 restrictions. Sustainable implementation of the intervention was rated as “possible” for 66.7% of the intervention components (n = 8) in Setting A, 63.6% (n = 7) in Setting B, and 0% (n = 0) in Setting C; it was rated as “not possible” for 8.3% of the intervention components (n = 1) in Setting A, 0% (n = 0) in Setting B, and 66.7% (n = 4) in Setting C. For a few intervention components, it was not possible to draw absolute conclusions regarding their implementation status or expected sustainability due to missing information from participants or inconclusive responses (i.e., agreement rate below 66.7%), leaving some percentages. The results of the appraisal of the intervention components regarding the creation of new capabilities, their effectiveness, their fit to the end-users and setting, and their perceived value within the organization can be found in Supplementary material 4.

Key factors: Influence on the success or failures of CP

Contextual factors

Qualitative content analysis of the interview data revealed the following 11 different subjects for the category of key factors influencing intervention implementation: champion, commitment, embedment, empowerment, engagement, health-promoting leadership, ownership, relevance, resources, responsibility, and strategic planning (for more details, see Table 2). Eight of these key factors, that is, commitment, embedment, engagement, health-promoting leadership, ownership, relevance, resources, and strategic planning, were identified in all settings; the other three key factors were each found in two of the three settings. Whether these key factors had a positive or negative influence on intervention implementation depended on the reported availability or unavailability within the settings. For example, the availability of commitment, engagement, health-promoting leadership, ownership, and strategic planning in Settings A and B had a beneficial effect on intervention implementation, whereas the unavailability of these key factors hindered the intervention implementation in Setting C. Furthermore, the presence of a champion who is devoted to the project and manages it with enthusiasm facilitated intervention implementation in Setting A, whereas the non-presence of this very champion had a hindering effect on intervention implementation in Setting C; in Setting B, this key factor was not mentioned. Moreover, some key factors need to be considered in a more differentiated way because they both facilitated and hindered intervention implementation, such as resources in Settings A and C. For example, a lack of personal resources had a negative influence on intervention implementation, while the provision of financial resources had a beneficial effect. Examining the key factors with respect to intervention implementation in the different settings, successful intervention implementation was associated with a higher number of available key factors, with n = 10 key factors in Setting A, n = 9 in Setting B, and n = 2 in Setting C. Conversely, a high number of unavailable key factors were found in Setting C (n = 9), resulting in failed intervention implementation.

TABLE 2

Table 2. Key factors influencing intervention implementation identified through qualitative content analysis.

During qualitative content analysis, we were able to extract not only the identified key factors, but also their reported effects and/or causes, thus establishing causal relationships. While the reported effects of the key factors were always associated with (un)successful intervention implementation, there were a variety of reported causes affecting the identified key factors. Hence, these causes behind the key factors can be referred to as the preceding factors. In contrast to the key factors, which showed high homogeneity across settings, the preceding factors were highly setting-specific. An example of a reported causal relationship between the preceding factor and key factor was the positive attitude toward PA, leading to high levels of engagement in Settings A and B, which, in turn, was a key factor facilitating intervention implementation. In Setting C, the lack of a positive attitude toward PA led to a low level of commitment, which emerged as a key factor that hindered intervention implementation. Furthermore, COVID-19 pandemic and personnel changes were found to be preceding factors in all three settings. Although these challenges were largely overcome through strategic planning in Settings A and B, they resulted in a missing champion, low commitment, low engagement, low relevance, lack of responsibility, and lack of strategic planning in Setting C. In addition to the identified causal relationships between the key factors and preceding factors, causal relationships were also revealed between the key factors themselves. For example, health-promoting leadership influenced the provision of resources in all three settings; while health-promoting leadership facilitated the provision of resources in Settings A and B, the provision of resources was deficient because of the lack of health-promoting leadership in Setting C. A detailed overview of all identified key factors and preceding factors, including their reported causal relationships, is illustrated in Figure 4.

FIGURE 4

Figure 4. Reported causal relationships of key factors and preceding factors influencing intervention implementation identified through qualitative content analysis. The causal loop diagrams were produced using Kumu Inc (retrieved from https://kumu.io/).

The analysis of the CP questionnaire data on the predefined factors influencing CP yielded some significant differences across settings (see Supplementary material 2), underscoring the differences in the key factors identified in the interviews. For example, significant differences between settings were found for ownership Item 1 (H(2) = 10.37, p = 0.006) and empowerment Item 2 (H(2) = 7.08, p = 0.029), both with higher scores for Setting A compared with Setting C, while higher scores for Setting A compared with Setting B were found for engagement Item 2 (H(2) = 6.30, p = 0.043). As another predefined factor influencing CP, we also found differences in organizational readiness across settings. We observed significantly higher scores in total ORIC (H(2) = 7.83, p = 0.020) and Change Efficacy (H(2) = 9.00, p = 0.011) for Setting B compared with Setting C (see Supplementary material 5). No significant differences were found in the Change Commitment scores across all settings (H(2) = 4.61, p = 0.100).

Discussion

What is the key to successful intervention implementation?

The current study contributes to a better understanding of CP as a co-creation approach for promoting a physically active lifestyle by answering questions about (1) the transferability of CP and (2) the associated key factors influencing its success or failure, particularly intervention implementation. Overall, the transfer of CP to new settings in the nursing care and automotive mechatronic sectors was realized, though the achieved impact varied by setting. Comparing the results of the three settings, CP resulted in the development and implementation of intervention components in Settings A and B, whereas in Setting C, a multi-component intervention was developed but not implemented. In this context, 11 key factors influencing intervention implementation were identified: champion, commitment, embedment, empowerment, engagement, health-promoting leadership, ownership, relevance, resources, responsibility, and strategic planning. The identified key factors are confirmed by the implementation science literature in general (48, 49) and in the specific settings of schools (50, 51) and workplaces (52, 53). Moreover, these key factors show a high overlap with the contextual factors that we have predefined based on previous research (36).

It is striking that the key factors identified were very similar across the three settings, but the different manifestations of these factors seem to determine the implementation or non-implementation of interventions. Thus, the presence of numerous key factors in Settings A and B resulted in the implementation of interventions, whereas the absence of these factors led to the lack of intervention implementation in Setting C. In addition to the key factors, we identified preceding factors that had an impact on these very key factors and, thus, indirectly influenced intervention implementation. These preceding factors were characterized by a high degree of setting specificity. However, some of these factors are consistent with influencing factors reported in the implementation science literature, such as personnel changes, political support, and qualification in the school setting (50, 51) or intraorganizational changes, personnel changes, and support in the workplace setting (52, 53).

The role of co-creation

By triangulating the quantitative and qualitative findings, we were able to uncover the relationships between activity characteristics and key factors. More precisely, some of the identified effects of activity characteristics corresponded to the identified key factors influencing intervention implementation. Thus, these activity characteristics seem to have had an impact on the manifestation of key factors, thereby also influencing intervention implementation.

In Setting A, for instance, all observed activity characteristics resulted in positive effects. For example, the participation of relevant stakeholders led to an increased commitment to and relevance of PA and health. In addition, both commitment and relevance were identified as key factors contributing to successful intervention implementation. The successful involvement of stakeholders was also supported by the results of the questionnaire survey, which showed high ratings for the subscale stakeholders in Setting A. The positive impact of stakeholder participation on commitment and relevance has also been the subject of other research articles (9, 12), indicating that partnerships between researchers and non-academic stakeholders are a promising approach for translating research findings into practice.

In comparison, in Setting B, not only the positive but also the negative effects of activity characteristics were found. For example, the group composition resulted in low engagement and a missing definition of responsibilities. These two effects of the group composition were also identified as key factors: engagement both facilitated and hindered intervention implementation, while responsibility facilitated intervention implementation. What might seem contradictory at first sight is a good example of the complexity of such processes and interactions of activity characteristics, key factors, and outcomes. For example, challenges can arise, while other factors simultaneously contribute to overcoming barriers (54), as the current study has uncovered in Setting B.

In Setting C, on the contrary, the observed activity characteristics mainly had negative effects. In this context, group composition appeared to be the most challenging, with negative effects on health-promoting leadership, ownership, and responsibility, all of which were identified as key factors and, thus, contributing to the failure of intervention implementation. The challenges associated with the group composition may have been caused by the lack of leadership participation, as well as the great heterogeneity of the involved practitioners (see the results of the structured minutes). More specifically, the lack of leadership participation may have hindered the decision-making process (see the preceding factor decision making, showing a lack of stakeholders' decision-making power). This is consistent with the findings from Nguyen et al. (55), emphasizing the importance of including decision- or policy-makers to achieve impact and implement the findings for integrated knowledge translation processes. Moreover, the great heterogeneity among practitioners may have increased the competing interests, which may have complicated the definition and adoption of responsibilities for intervention implementation. This illustrates a dilemma of co-creation because all relevant stakeholders should be involved (2), but at the same time, this increases the risk of conflicts arising from differing interests and perspectives (11, 35, 56).

Notably, the activity characteristic researchers' input & support had a positive effect on the relevance of the project in all three settings, underlining the importance of the researchers' role and contribution in the planning phase. This was supported by other studies highlighting the involvement of researchers as a key performance indicator for enhancing CP (19) and recommending that researchers work closely with end-users and other non-academic stakeholders from the outset of a co-creation process to ensure the relevance of findings (57). Overall, the current study highlights the complex and setting-specific interplay between activity characteristics and key factors, as well as the relevance of activity characteristics for the success or failure of the intervention implementation.

Fit of co-creation approaches

The findings suggest that some settings might benefit more from a co-creation approach for PA promotion than others, with more favorable effects in the nursing care setting than in the automotive mechatronics setting. This may question a co-creation approach as a panacea leading to successful intervention implementation. Here, it might be advisable to consider in advance whether or, in particular, how the use of a co-creation approach is appropriate for a particular setting.

A first starting point to determine the fit of a co-creation approach can be the readiness for a change (40), such as PA promotion, in a specific setting. In the present study, we examined organizational readiness as a predefined factor influencing CP, here as operationalized by Shea et al. (41); our results failed to reveal that higher change efficacy and commitment comes with a more successful CP process. A recent review by Miake-Lye et al. (58) has shown that this organizational readiness assessment covers mainly the construct “readiness for implementation” as it is used in the Consolidated Framework for Implementation Research (CFIR; domain “inner setting”) (49). Concurrently, other organizational readiness for change assessments [e.g., (59, 60)] cover far more CFIR constructs (e.g., domains “characteristics of individuals,” “process”) (58). In this context, it may be important to consider more setting-specific information to classify a setting using a readiness scale to predict an organization's ability to conduct a change. However, implementation and especially determinant frameworks include relevant constructs and can be useful for mapping and developing a comprehensive organizational readiness instrument (58, 60, 61). For a more setting-specific application of the organizational readiness concept, the key factors of intervention implementation as identified in our multiple case study might also be useful for a readiness assessment. This readiness judgment should then be followed by a recommendation of strategies to enhance readiness before a co-creation process is conducted, for example, by identifying and preparing a champion [see the typology of readiness development strategies by Vax et al. (62)].

Second, classifying a setting as ready for change may not necessarily mean that this setting is also ready to engage in a co-creation process. Since participation is a core element of co-creation, a setting's readiness for participation, in which stakeholders' participation is considered important and valuable, is crucial for conducting a co-creation process. Vice versa, a setting completely closed to the stakeholders' participation may be unsuitable for a co-creation process (63). Moreover, participating in a co-creation process is not without costs for stakeholders because stakeholders' willingness and opportunities to invest additional resources are major requirements for conducting a co-creation process. Conversely, less emphasis may be placed on using a co-creation process when time or resources are limited (35). To determine a setting's readiness for participation, it might be useful to evaluate this readiness within the scope of an organizational readiness assessment, as done by Robertson et al. (64). This information can then be used to decide whether a co-creation approach seems suitable in a setting, prerequisites first need to be created (e.g., provision of resources), or another approach, such as implementing researcher-developed interventions, would be more appropriate.

Finally, a co-creation approach with the aim of PA promotion should be tailored to the unique needs and opportunities of the setting. This was supported by recent research emphasizing that co-creation is largely context-dependent (32, 57, 65), highlighting the need for localized solutions not only for the development of tailored interventions, but also for the realization of a co-creation process itself to account for the uniqueness of settings. Thus, a setting-specific selection of co-creation steps and principles or potential adaptations may be required to achieve an optimal fit between the chosen co-creation approach and given setting. In this regard, there is a growing body of literature focusing on providing guidance for the design of co-creation processes. For example, principles and strategies for partnerships with researchers and stakeholders (32, 66), or an instrument to help researchers select the appropriate tools to foster the impact of co-creation processes (67) are provided.

Strengths and limitations

The current comprehensive mixed methods evaluation embedded in a multiple case study allowed us to gain new insights into the “black box” transferability and key factors of CP. Given the heterogeneity and flexibility of co-creation processes, this design was found to be appropriate for examining our research questions within and between three settings. In particular, the qualitative content analysis by extraction was a major strength because it enabled us not only to identify important activity characteristics and key factors, but also to determine the causal relationships between them and their reported causes and/or effects. This has given us a deep understanding of the dynamics and complexity of how these factors interact in the respective settings.

However, some limitations must be considered. First, as outlined in the study protocol, the measurement of outcomes at the individual level (i.e., PA behavior, PAHCO, health status) was planned in a pre-post design but finally not possible, as practitioners self-initiated the implementation of intervention components at an early stage (36). Therefore, in examining the transfer of CP, we refer to the logic model components Activities, Outputs and Structural outcomes. Second, the findings of the ORIC questionnaire should be interpreted with caution, as only a small sample size was reached, mainly because only people who participated in the first planning meeting took part in the survey. Third, we had a moderate response rate to the request for participation in the CP questionnaires; thus, not all the perspectives of the stakeholders on the organization and realization of planning meetings, the current implementation status and appraisal of interventions, and predefined factors influencing CP may be represented. However, we aimed to obtain missing information and gain deeper insights into the transferability and key factors of CP in different settings, here by conducting additional interviews and selecting interviewees through a purposeful sampling of information-rich cases. Fourth, the interview guide was pilot tested only within the research team, and the transcripts and findings were not returned to the interviewees for comments and feedback. Fifth, the identified causal relationships only refer to the interviewees' qualitative reports.

Conclusion

The present article contributes to a better understanding of a co-creation approach utilized for PA promotion by providing new insights into (1) the transferability of CP as a co-creation approach and (2) the associated key factors influencing its success or failure, particularly intervention implementation. Specifically, the in-depth mixed methods evaluation in three settings in the nursing care and automotive mechatronic sectors provided relevant findings for future research. As a main result, transferring CP to new settings was achieved, though differences between the three settings were identified and demonstrated. Particularly, the achieved impact of CP varied by setting: while CP resulted in the development and implementation of PA-promoting interventions in nursing care settings, a multi-component intervention was developed but not implemented in the automotive mechatronics setting. In this context, we identified multiple key factors influencing intervention implementation and, thus, the success or failure of CP. These key factors also varied by setting, interacted in a complex way, and were related to co-creation activities. Therefore, future co-creation initiatives should carefully consider the specific characteristics of a setting to determine whether it is truly ready to initiate a change, such as PA promotion, and ready to engage in a co-creation process. Moreover, future research should investigate the complex and dynamic interactions between key factors to generate a theoretical foundation for the implementation and evaluation of such processes.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Ethical Committee of the Friedrich-Alexander-Universität Erlangen-Nürnberg. The patients/participants provided their written informed consent to participate in this study.

Author contributions

JP and EG conducted the qualitative and quantitative data collection, analyzed the qualitative and quantitative data, and drafted the manuscript. JC supported the qualitative and quantitative data collections and analyses. KP and JS acquired funding. KP supervised the work. All authors were involved in designing the study, interpreting data, critically reviewing drafts of the manuscript, and reading and approving the final manuscript.

Funding

This research was conducted as part of the Physical Activity-related Health Competence in Apprenticeship and Vocational Education (PArC-AVE) project with the associated research network Capital4Health, which was funded by the German Federal Ministry of Education and Research (Grant No. 01EL1821A). The funder had no role in study design, data collection and analysis, decision to publish, and preparation of the manuscript.

Acknowledgments

We thank Laura Gabel, Elisabeth Geyer, and Eva Lorenz for their support in data collection, analyses, and preparation.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2022.975638/full#supplementary-material

References

1. McConnell T, Best P, Davidson G, McEneaney T, Cantrell C, Tully M. Coproduction for feasibility and pilot randomized controlled trials: learning outcomes for community partners, service users and the research team. Res Involv Engagem. (2018) 4:32. doi: 10.1186/s40900-018-0116-0

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Leask CF, Sandlund M, Skelton DA, Altenburg TM, Cardon G, Chinapaw MJ, et al. Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Res Involv Engagem. (2019) 5:2. doi: 10.1186/s40900-018-0136-9

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Borek AJ, McDonald B, Fredlund M, Bjornstad G, Logan S, Morris C. Healthy Parent Carers programme: development and feasibility of a novel group-based health-promotion intervention. BMC Public Health. (2018) 18:270. doi: 10.1186/s12889-018-5168-4

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Giné-Garriga M, Dall PM, Sandlund M, Jerez-Roig J, Chastin SF, Skelton DA, et al. Pilot randomised clinical trial of a novel approach to reduce sedentary behaviour in care home residents: feasibility and preliminary effects of the GET READY study. Int J Environ Res Public Health. (2020) 17:2866. doi: 10.3390/ijerph17082866

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Buckley BJ, Thijssen DH, Murphy RC, Graves LE, Cochrane M, Gillison F, et al. Pragmatic evaluation of a coproduced physical activity referral scheme: a UK quasi-experimental study. BMJ Open. (2020) 10:e034580. doi: 10.1136/bmjopen-2019-034580

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Halvorsrud K, Kucharska J, Adlington K, Rüdell K, Brown Hajdukova E, Nazroo J, et al. Identifying evidence of effectiveness in the co-creation of research: a systematic review and meta-analysis of the international healthcare literature. J Public Health (Oxf). (2021) 43:197–208. doi: 10.1093/pubmed/fdz126

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of Community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. (1998) 19:173–202.

PubMed Abstract | Google Scholar

8. Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. (2010) 100:S40–6. doi: 10.1186/1748-5908-4-50

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Cargo M, Mercer SL. The value and challenges of participatory research: strengthening its practice. Annu Rev Public Health. (2008) 29:325–50. doi: 10.1146/annurev.publhealth.29.091307.083824

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice, and policy. Health Educ Res. (1998) 13:87–108.

PubMed Abstract | Google Scholar

11. Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. (2016) 94:392–429. doi: 10.1111/1468-0009.12197

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q. (2012) 90:311–46. doi: 10.1111/j.1468-0009.2012.00665.x

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Bodkin A, Hakimi S. Sustainable by design: a systematic review of factors for health promotion program sustainability. BMC Public Health. (2020) 20:964. doi: 10.1186/s12889-020-09091-9

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Holtermann A, Krause N, van der Beek AJ, Straker L. The physical activity paradox: six reasons why occupational physical activity (OPA) does not confer the cardiovascular health benefits that leisure time physical activity does. Br J Sports Med. (2018) 52:149–50. doi: 10.1136/bjsports-2017-097965

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Prince SA, Rasmussen CL, Biswas A, Holtermann A, Aulakh T, Merucci K, et al. The effect of leisure time physical activity and sedentary behavior on the health of workers with different occupational physical activity demands: a systematic review. Int J Behav Nutr Phys Act. (2021) 18:100. doi: 10.1186/s12966-021-01166-z

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Carl J, Grüne E, Popp J, Pfeifer K. Physical activity promotion for apprentices in nursing care and automotive mechatronics – competence counts more than volume. Int J Environ Res Public Health. (2020) 17:793. doi: 10.3390/ijerph17030793

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Rütten A. Kooperative Planung und Gesundheitsförderung: Ein Implementationsansatz. Zeitschrift für Gesundheitswissenschaft. (1997) 5:257–72.

Google Scholar

18. Rütten A. Evaluating healthy public policies in community and regional contexts. In: Rootman I, Goodstadt M, Hyndman B, McQueen DV, Potvin L, Springett J, et al., editors. Evaluation in Health Promotion: Principles and Perspectives. Geneva: World Health Organization Europe (2001). p. 341–64.

PubMed Abstract | Google Scholar

19. Sommer R, Linder S, Ziemainz H, Gelius P. Key performance indicators of cooperative planning processes: case study results from German sport science and physical activity promotion projects. German J Exe Sport Res. (2022) 52:24–38. doi: 10.1007/s12662-021-00745-3

CrossRef Full Text | Google Scholar

20. Gelius PC, Jansen M, King AC. Cooperative planning and its utilization in German physical activity promotion: a brief introduction. Health Promot Int. (2021) 36:ii1–7. doi: 10.1093/heapro/daab170

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Sudeck G, Pfeifer K. Physical activity-related health competence as an integrative objective in exercise therapy and health sports – conception and validation of a short questionnaire. Sportwiss. (2016) 46:74–87. doi: 10.1007/s12662-016-0405-4

CrossRef Full Text | Google Scholar

22. Carl J, Sudeck G, Pfeifer K. Competencies for a healthy physically active lifestyle-reflections on the model of physical activity-related health competence. J Phys Activity Health. (2020) 17:688–97. doi: 10.1123/jpah.2019-0442

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Popp J, Carl J, Grüne E, Semrau J, Gelius P, Pfeifer K. Physical activity promotion in German vocational education: does capacity building work? Health Promot Int. (2020) 35:1577–89. doi: 10.1093/heapro/daaa014

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Grüne E, Popp J, Carl J, Semrau J, Pfeifer K. Examining the sustainability and effectiveness of co-created physical activity interventions in vocational education and training: a multimethod evaluation. BMC Public Health. (2022) 22:765. doi: 10.1186/s12889-022-13133-9

PubMed Abstract | CrossRef Full Text | Google Scholar

25. World Health Organization and ExpandNet. Nine Steps For Developing a Scaling-Up Strategy. Geneva: World Health Organization (2010).

Google Scholar

26. Reis RS, Salvo D, Ogilvie D, Lambert EV, Goenka S, Brownson RC. Scaling up physical activity interventions worldwide: stepping up to larger and smarter approaches to get people moving. Lancet. (2016) 388:1337–48. doi: 10.1016/S0140-6736(16)30728-0

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Wang S, Moss JR, Hiller JE. Applicability and transferability of interventions in evidence-based public health. Health Promot Int. (2006) 21:76–83. doi: 10.1093/heapro/dai025

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Frahsa A, Rütten A, Roeger U, Abu-Omar K, Schow D. Enabling the powerful? Participatory action research with local policymakers and professionals for physical activity promotion with women in difficult life situations. Health Promot Int. (2014) 29:171–84. doi: 10.1093/heapro/das050

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Rütten A, Wolff A, Streber A. Interaktiver Wissenstransfer in der Gesundheitsförderung: das GESTALT-Projekt. Erste Ergebnisse der Erprobung eines Ansatzes zur nachhaltigen Implementierung evidenzbasierter Bewegungsprogramme Gesundheitswesen. (2016) 78:359–66. doi: 10.1055/s-0035-1548882

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Loss J, Brew-Sam N, Metz B, Strobl H, Sauter A, Tittlbach S. Capacity building in community stakeholder groups for increasing physical activity: results of a qualitative study in two German communities. Int J Environ Res Public Health. (2020) 17:2306. doi: 10.3390/ijerph17072306

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Strobl H, Ptack K, Töpfer C, Sygusch R, Tittlbach S. Effects of a participatory school-based intervention on students' health-related knowledge and understanding. Front Public Health. (2020) 8:122. doi: 10.3389/fpubh.2020.00122

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Hoekstra F, Mrklas KJ, Khan M, McKay RC, Vis-Dunbar M, Sibley KM, et al. A review of reviews on principles, strategies, outcomes and impacts of research partnerships approaches: a first step in synthesising the research partnership literature. Health Res Policy Syst. (2020) 18:51. doi: 10.1186/s12961-020-0544-9

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Verloigne M, Altenburg TM, Chinapaw MJ, Chastin S, Cardon G, De Bourdeaudhuij I. Using a co-creational approach to develop, implement and evaluate an intervention to promote physical activity in adolescent girls from vocational and technical schools: a case control study. Int J Environ Res Public Health. (2017) 14:862. doi: 10.3390/ijerph14080862

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Flinders M, Wood M, Cunningham M. The politics of co-production: risks, limits and pollution. evid policy. (2016) 12:261–79. doi: 10.1332/174426415X14412037949967

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Oliver K, Kothari A, Mays N. The dark side of coproduction: do the costs outweigh the benefits for health research? Health Res Policy Syst. (2019) 17:33. doi: 10.1186/s12961-019-0432-3

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Popp J, Grüne E, Carl J, Semrau J, Pfeifer K. Co-creating physical activity interventions: a mixed methods evaluation approach. Health Res Policy Syst. (2021) 19:37. doi: 10.1186/s12961-021-00699-w

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Crane M, Bauman A, Lloyd B, McGill B, Rissel C, Grunseit A. Applying pragmatic approaches to complex program evaluation: a case study of implementation of the New South Wales Get Healthy at Work program. Health Promot J Austr. (2019) 30:422–32. doi: 10.1002/hpja.239

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Johnson RB, Onwuegbuzie AJ, Turner LA. Toward a Definition of Mixed Methods Research. J Mix Methods Res. (2007) 1:112–33. doi: 10.1177/1558689806298224

CrossRef Full Text | Google Scholar

39. Patton MQ. Qualitative Research And Evaluation Methods. Los Angeles: Sage (2015).

Google Scholar

40. Weiner BJ.A theory of organizational readiness for change. Implement Sci. (2009) 4:67. doi: 10.1186/1748-5908-4-67

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implementation Science. (2014) 9:7. doi: 10.1186/1748-5908-9-7

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Kuckartz U. Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterstützung Qualitative Content Analysis: Methods, Practice, Computer Support. Weinheim, Basel: Beltz Juventa (2016).

Google Scholar

43. Gläser J, Laudel G. Experteninterviews Und Qualitative Inhaltsanalyse Als Instrumente Rekonstruierender Untersuchungen. Wiesbaden: VS Verlag (2010).

Google Scholar

44. Gläser J, Laudel G. Life with and without coding: two methods for early-stage data analysis in qualitative research aiming at causal explanations. Forum Qualit Sozialforschung / Forum Qualitative Soc Res. (2013).

Google Scholar

45. Laudel G. MIA (2019). Available online at: http://www.laudel.info/downloads/mia/ (accessed August 05, 2021).

Google Scholar

46. O'Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. (2010) 341:c4587. doi: 10.1136/bmj.c4587

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Farmer T, Robinson K, Elliott SJ, Eyles J. Developing and implementing a triangulation protocol for qualitative health research. Qual Health Res. (2006) 16:377–94. doi: 10.1177/1049732305285708

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. (2008) 41:327–50. doi: 10.1007/s10464-008-9165-0

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4:50.

PubMed Abstract | Google Scholar

50. Cassar S, Salmon J, Timperio A, Naylor P-J, van Nassau F, Contardo Ayala AM, et al. Adoption, implementation and sustainability of school-based physical activity and sedentary behavior interventions in real-world settings: a systematic review. Int J Behav Nutr Phys Act. (2019) 16:120. doi: 10.1186/s12966-019-0876-4

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Naylor P-J, Nettlefold L, Race D, Hoy C, Ashe MC, Wharf Higgins J, et al. Implementation of school based physical activity interventions: a systematic review. Prev Med. (2015) 72:95–115. doi: 10.1016/j.ypmed.2014.12.034

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Rojatz D, Merchant A, Nitsch M. Factors influencing workplace health promotion intervention: a qualitative systematic review. Health Promot Int. (2017) 32:831–9. doi: 10.1093/heapro/daw015

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Wierenga D, Engbers LH, van Empelen P, Duijts S, Hildebrandt VH, van Mechelen W. What is actually measured in process evaluations for worksite health promotion programs: a systematic review. BMC Public Health. (2013) 13:1190. doi: 10.1186/1471-2458-13-1190

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implementat Sci. (2016) 11:20. doi: 10.1186/s13012-016-0383-9

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Nguyen T, Graham ID, Mrklas KJ, Bowen S, Cargo M, Estabrooks CA, et al. How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field. Health Res Policy Sys. (2020) 18:35. doi: 10.1186/s12961-020-0539-6

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Jansen MW, van Oers H, Kok G, de Vries NK. Public health: disconnections between policy, practice, and research. Health Res Policy Syst. (2010) 8:37. doi: 10.1186/1478-4505-8-37

PubMed Abstract | CrossRef Full Text | Google Scholar

57. van der Graaf P, Cheetham M, Redgate S, Humble C, Adamson A. Co-production in local government: process, codification and capacity building of new knowledge in collective reflection spaces. Workshops findings from a UK mixed methods study. Health Res Policy Syst. (2021) 19:12. doi: 10.1186/s12961-021-00677-2

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Miake-Lye IM, Delevan DM, Ganz DA, Mittman BS, Finley EP. Unpacking organizational readiness for change: an updated systematic review and content analysis of assessments. BMC Health Serv Res. (2020) 20:106. doi: 10.1186/s12913-020-4926-z

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Lehman WE, Greener JM, Simpson D. Assessing organizational readiness for change. J Subst Abuse Treat. (2002) 22:197–209. doi: 10.1016/S0740-5472(02)00233-7

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Helfrich CD Li Y-F, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the promoting action on research in health services (PARIHS) framework. Implement Sci. (2009) 4:38. doi: 10.1186/1748-5908-4-38

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Serhal E, Arena A, Sockalingam S, Mohri L, Crawford A. Adapting the consolidated framework for implementation research to create organizational readiness and implementation tools for project ECHO. J Contin Educ Health Prof. (2018) 38:145–51. doi: 10.1097/CEH.0000000000000195

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Vax S, Farkas M, Russinova Z, Mueser KT, Drainoni ML. Enhancing organizational readiness for implementation: constructing a typology of readiness-development strategies using a modified Delphi process. Implement Sci. (2021) 16:61. doi: 10.1186/s13012-021-01132-0

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Zhang Y, Flum M, West C, Punnett L. Assessing organizational readiness for a participatory occupational health/health promotion intervention in skilled nursing facilities. Health Promot Pract. (2015) 16:724–32. doi: 10.1177/1524839915573945

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Robertson MM, Tubbs D, Henning RA, Nobrega S, Calvo A, Murphy LA. Assessment of organizational readiness for participatory occupational safety, health and well-being programs. Work. (2021) 69:1317–42. doi: 10.3233/WOR-213552

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Williams O, Sarre S, Papoulias SC, Knowles S, Robert G, Beresford P, et al. Lost in the shadows: reflections on the dark side of co-production. Health Res Policy Syst. (2020) 18:43. doi: 10.1186/s12961-020-00558-0

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Smith B, Williams O, Bone L, the Moving Social Work Co-production Collective. Co-production: a resource to guide co-producing research in the sport, exercise, and health sciences. Qual Res Sport Exe Health. (2022):1–29. doi: 10.1080/2159676X.2022.2052946

CrossRef Full Text | Google Scholar

67. van Dijk-de Vries A, Stevens A, van der Weijden T, Beurskens AJ. How to support a co-creative research approach in order to foster impact. The development of a Co-creation Impact Compass for healthcare researchers. PLoS ONE. (2020) 15:e0240543. doi: 10.1371/journal.pone.0240543

PubMed Abstract | CrossRef Full Text | Google Scholar