Tiana Felmingham1,2*
Kathryn Backholer1,2Elizabeth Hoban2
Andrew D. Brown1,2Phoebe Nagorcka-Smith1,2
Steven Allender1,2- 1Global Centre for Preventive Health and Nutrition, Institute for Health Transformation, Deakin University, Geelong, VIC, Australia
- 2School of Health and Social Development, Faculty of Health, Deakin University, Geelong, VIC, Australia
Systems thinking approaches are increasingly being used to help communities understand and develop responses to preventing complex health problems. Less is known about how success is characterized and what influences success in these approaches. We present a systematic review of how concepts of success are understood and evaluated in the peer reviewed literature of studies using systems thinking in community prevention. We searched five databases for peer-reviewed literature published between 2000 and 2022, with search terms related to systems thinking, prevention and community. Studies were included if they; reported using community-based systems thinking to prevent a public health problem; described the engagement and empowerment of community members to address a public health issue; and, were published in English. Thirty-four articles were identified from 10 countries. Twenty-one aimed to prevent a chronic disease (e.g., obesity) and 16 measured success using specific tools, 10 of which used semi-structured interviews or surveys. Measures of success included implementation processes, cultural appropriateness, the number or type of actions implemented, effectiveness of community action, and changes in individual thinking or mental models, population health outcomes, data collected, or systems level measures. Implementation factors influencing success included the capacity to engage participants, composition and experience of facilitators, strength of coordination teams, allocation of resources, adaptation to participant feedback, use of multiple systems approaches, workshop process providing time and methods to allow new insights, flexible delivery, and diversity of perspectives. Findings from each of the articles indicated that approaches increased a range of outcomes including community action, strategic thinking, future planning and evaluation, community buy-in, community voice, contribution and leadership, in addition to developing shared visions and goals and creating new, ongoing collaborations, among many others. Measures of success varied, suggesting more empirical reporting of proposed outcomes of system science in communities would be valuable. While the measurement of success in the use of systems thinking in community-based prevention efforts is limited, there are helpful examples we can look to for future measurement of success.
1. Introduction
The application of systems thinking to address complex community and social problems is gaining momentum (1, 2), particularly in community settings (3, 4). A range of methods have emerged within systems thinking that are used to help capture and engage with complexity inherent in many modern problems (5). According to Ison (p. 142) (5), systems thinking (or systemic thinking) in this instance is considered to be ‘the understanding of a phenomenon within the content of a larger whole; to understand things systemically literally means to put them into context, to establish the nature of their relationships.’
There are a wide range of systems thinking approaches, which are shaped by the various historical influences of systems practice across different disciplines (6). Common techniques to work with communities using system thinking include participatory system dynamics (PSD), group model building (GMB), soft systems methodology (7), critical systems heuristics (8) and community-based system dynamics (CBSD) (9, 10). While there is overlap between methods, there are also key distinctions, which generally span the level of involvement participants have in the process, the ownership participants have over the diagram developed and overall capacity built as a result of participant engagement (10). Most examples of systems thinking studies in the public health literature provide in-depth descriptions of the community’s understanding of a complex problem, but few provide insights on the effectiveness of the method, nor the implications of these methods for the success of attempts to address the problem overall. GMB stands out as one form of systems thinking with a greater amount of documented evaluation in the literature (3, 11, 12).
Long before systems thinking gained momentum in public health, community participation and engagement have been called for as a critical element in prevention efforts (13, 14). CBSD is an application of GMB that emphasizes participation and engagement alongside systems thinking (10). A key aspect of CBSD is engaging community or stakeholders in an agreed problem to gain shared insights and identify corresponding community-led action through the use of GMB (9, 10). This typically involves stakeholders in a series of workshops or consultations who create a diagram (in public health, often a causal loop diagram (CLD)) which helps visualize a complex problem from the community’s perspective. CBSD builds community capacity to recognize key feedback loops in a system’s structure that drive a system’s behavior, mobilizing action for systems change.
The concept of success can be contentious, and for the purpose of this review, success (or not) of an approach is considered in light of the authors conclusions within each article. While an approach is not considered completely ‘successful’ or ‘unsuccessful’, it is important to draw from past experiences that may have included components that helped facilitators get closer to their outcome, or those that may have created challenges.
While there are numerous descriptions of the use of CBSD to identify causal factors, interrelationships and actions to address a problem, much less literature describes the effectiveness or success of the approach. Within the literature that is available, findings are fragmented. No study has systematically searched the literature to examine success of CBSD across multiple studies, nor identified factors that influence success.
This systematic review assesses the current evidence describing success of CBSD and examines implementation factors that influence this success by asking the following research questions:
1. How is success in community-based system dynamics understood and measured in public health?
2. What implementation factors influence success of community-based system dynamics efforts in public health?
2. Methods
This review was registered with PROSPERO in January 2021 (CRD42021212817). Reporting of results was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (15).
Our review focused on systems thinking approaches that specifically brought the community together to address the prevention of a public health or social problem, specifically using CBSD, or where a participatory method of visualization, modeling or causal diagram creation was applied to empower or mobilize a community in response to a complex problem. Definitions of CBSD by Hovmand (9) and descriptions by Király and Miskolczi (10) have been used to define the boundaries of this review.
2.1. Search strategy
The search was inclusive of empirical research published between January 2000 and October 2022. Both qualitative and quantitative study designs were included in our review. Only articles published in English were included.
Studies were searched using the MEDLINE complete, PsycInfo, CINAHL, Global Health and SocIndex databases. Search terms focused on three primary areas: community (population), systems thinking (intervention), prevention (outcome). Terms from the three areas were combined with the operator ‘AND’. Within the primary search areas, more specific search terms were combined with the operator “OR” (Table 1). The broad term of “systems thinking” was included as pilot literature searching identified there were few published studies that measure success of CBSD when using these terms alone.
Table 1. Search term concepts and variations.
The search strategy was adapted to the syntax requirements of each database. Reference lists of all included articles, and other relevant review articles identified, were additionally scanned for relevant studies.
All retrieved references were exported into the Endnote reference management software and transferred to Covidence, an online review platform, where duplicates were removed and articles were screened for inclusion.
2.2. Study selection
Use of CBSD terminology is sporadic. For the purpose of this paper, we will use the term CBSD when describing success, implementation and measures for all included articles in our review, even if the term has not been stated in the included article. This provides recognition of those articles using methods that encompass the principles of CBSD, as described in the following inclusion criteria.
Articles were included if they reported projects that; described collaboration or coalitions within specified communities; used, or described using an approach to systems thinking in the community setting (stated they were using CBSD, or alternatively, GMB, participatory systems or described building/using a qualitative CLD with the community); described engaging with stakeholders to apply systems thinking; focused on prevention of a public health issue; described a process that intended to empower individuals from a community (to take action, mobilize, or advocate); and had participation of community members across all stages of problem definition, diagram development, testing and transferring insights back into community. Studies were excluded if they did not consider community-level outcomes. An end point for the CBSD process was not defined in the criteria, as this varied and was highly dependent on what facilitators intended to see change as a result of using CBSD.
2.3. Screening process
The titles and abstracts of retrieved articles were independently screened by two members of the review team (TF, ADB or PNS) and discrepancies were resolved through discussion with a third reviewer (ADB or PNS). Before starting full text review, three authors involved in the screening process (TF, ADB, PNS) reviewed a subset of articles to ensure application of criteria were consistent. The remaining full text articles were reviewed by two independent authors (TF with ADB or PNS) with conflicts discussed and resolved between three authors (TF, ADB, PNS). Reasons for article exclusion at this stage were recorded. The most common reasons for exclusion were articles that were the wrong study type, wrong systems approach, or wrong health issue.
2.4. Data extraction and synthesis
Two reviewers (TF, ADB or PNS) independently extracted relevant data from 20% of articles (selected in alphabetical order, by first author) using a pre-specified and agreed upon data extraction template which included study title, intervention title (if specified), country or region of study/implementation, year of publication and implementation, author (s)/organization (s), study design, aim, nature of complex problem, lead implementation organization, collaborations, number and type of stakeholders involved in implementation (community members, professionals, others), details of the implementation process, method of data collection for success, and authors conclusions of the success of the process. Discrepancies were discussed within this sample to ensure consistency across the remainder of articles. Remaining data extraction was completed by one reviewer (TF).
The number of studies screened, assessed and included in the final review were recorded and reported using the PRISMA flowchart (Figure 1). As our study aimed to better understand the varying concepts of what constituted success in CBSD, a summary of the findings across the literature is presented (Supplementary Table 3). Results are grouped into studies that have used author observation to report on success and those that have used non-observation data collection methods.
Figure 1. PRISMA flow chart.
2.5. Quality assessment
We did not undertake an assessment for the risk of bias as this was a systematic review with narrative synthesis intended to summarize the current state of the literature. We did however apply a standard approach to assessing the quality of studies returned by the review.
Study quality was assessed using the Critical Appraisal Skills Programme (CASP) Checklists (16). Quality appraisal was conducted by the primary investigator (TF) with 20% of articles cross-checked by one member of the review team to ensure consistency (ADB or PNS). Although no literature was excluded based on quality, discussion on quality of studies is included.
2.6. Analysis
A narrative synthesis, guided by Popay and Roberts (17), was used in this review due to the varying nature of study designs employed by those using CBSD. This involved; developing a preliminary synthesis of findings of included studies, exploring relationships in the data, and assessing the robustness of the synthesis (for example, quality assessments and quantity, of included articles, and minimizing bias by including multiple reviewers at each stage).
Descriptions provided by authors of each article were used to identify interventions (if they were named), regions where approaches occurred, the nature of the complex problem identified, and how systems thinking was used with each community.
Inductive thematic analysis (18) was used by one reviewer (TF) to explore themes within three subsets of the data extracted, specifically where; data related to concepts of success, identification and descriptions of implementation factors, and, overall findings (for example, how an implementation factor increased or decreased perceived success). The reviewer (TF) identified codes and categories as they emerged, and where codes or categories were identified as similar, themes emerged. The main, reoccurring or most important concepts were identified across all included studies by identifying those that occurred most often or were described by authors as having a critical influence on results. Results for this analysis are presented in sections 3.5 and 3.6.
3. Results
The search of all databases yielded 6,825 articles, before 3,652 duplicates were removed. A total of 3,173 titles and abstracts were screened, with 284 articles identified eligible for full-text review. Of these, 34 articles met the inclusion criteria and were included in this review (Supplementary Table 3).
3.1. Sample characteristics
The 34 articles represented 12 different interventions. Twenty articles did not specify an intervention name. Two articles related to The Whole of Systems Trial of Prevention Strategies for Childhood Obesity (WHOSTOPS) (19, 20), with three additional articles focused on subset interventions within WHOSTOPS [GenR8 Change (21)], Sustainable Eating Activity Change Portland [or SEA Change Portland (22) and Portland, a WHO STOPS pilot community (23)]. Two articles related to Healthy Families Waitākere (HFW) (24, 25), and the remaining seven articles described a single intervention (Campbelltown - Changing our Future (Change4Campbelltown) (26), Nourishing Hawke’s Bay: He wairua tō te kai (27), Prevention Impacts Simulation Model (PRISM) (28), Shape Up Under 5 (SUU5) (29), Derby: a City on the Move (DaCotM) (30), Urban Health in Latin America (“Salud Urbana en América Latina,” or SALURBAL) (31), and the Food & Fitness (F&F) Initiative (32).
The articles reviewed were of mixed quality and value, based on the Critical Appraisal Skills Programme (CASP) Checklists (16) (Table 2). One article was assessed as low value (33), 23 moderately valuable (21, 23, 24, 27, 28, 30, 34–50) and 10 were of high value (19, 20, 22, 25, 26, 29, 31, 32, 51, 52).
Table 2. Summary of quality assessment for included articles.
3.2. Region or country
Articles described interventions that were implemented across 10 different countries or regions, with 26 articles describing interventions conducted within high income countries [13 from the United States (28, 29, 32, 35–37, 40, 44, 46–49, 52) with one article each related to PRISM and SUU5, with the remaining interventions not described, seven from Australia (19–23, 26, 51) with five articles connected to WHOSTOPS or subset interventions, one article related to the Change4Cambelltown intervention, and one intervention not described. There were three articles from New Zealand (24, 25, 27) two of which were related to the HFW intervention and one connected to Nourishing Hawke’s Bay: He wairua tō te kai. Three articles were from the United Kingdom (30, 34, 45)], one of which was connected to the DaCotM intervention, with the remaining two interventions not described. Four articles described interventions in upper middle income countries, in Lebanon (38, 39, 43) and Thailand (50), with all focused on refugee communities (it is worth noting that all interventions in Lebanon occurred in areas with high disadvantage, including those areas with newly arrived Syrian refugees, and the intervention in Thailand was related to those living in refugee camps), none of which identified a name for their interventions. One article described an intervention in Fiji (42), a middle income country, while two articles described interventions in lower middle income countries [one from India (33), one from the Latin American region (31)]. Of these, all were unnamed interventions, with the exception of the SALURBAL intervention in Latin America. One article described an unnamed intervention in Afghanistan (41), one of the world’s lowest income countries (53).
3.3. Complex problems
Of the articles included, 10 described interventions focused on childhood obesity (19–22, 27, 29, 40, 45, 48, 52), two on childhood fruit and vegetable intake (24, 25), and one on childhood overweight and obesity (26). The articles reported on findings from seven interventions: SUU5 (29), WHOSTOPS (19, 20), two additional interventions connected to WHOSTOPS (GenR8 Change (21) SEA Change Portland (22)), HFW (24, 25), Change4Campbelltown (26) and Nourishing Hawke’s Bay: He wairua tō te kai (27). Four articles that focused on childhood obesity described interventions that were unnamed (40, 45, 48, 52).
In addition, eight articles reported findings on CBSD approaches that focused on some aspect of chronic disease. This includes the following topics: chronic disease as an outcome (44), burden of chronic disease (28), non-communicable disease (43)), and changing environments to encourage physical activity and healthy eating (physical inactivity (30), availability of healthy foods in low income communities (36), use of evidence in food related policy making (42), water and sugar sweetened beverage consumption (23) and healthy eating and active living (32)) for the population overall.
Three articles reported on interventions that focused on mental health of refugee and local communities (38, 39, 41), all of which occurred in Lebanon and Afghanistan. Three articles focused on equity (racial inequity (47), health equity (31), and inequities experienced by Indigenous women in relation to intimate partner violence and alcohol misuse (46)). Two articles focused on housing (housing, energy and wellbeing (34) and family homeless shelter use (35)).
Other complex problems reported include; community violence (49), road traffic safety and pedestrian deaths (37) and sustained adoption of cleaner cooking technologies (33). One article did not describe the focus of interventions specifically, as its aim was to explore the use of CBSD in Indigenous communities in Australia across various interventions (51).
3.4. Use of systems thinking with the community
Twenty-six of the 34 articles used CBSD in the community in addition to testing or refining the systems method used. Four articles used CBSD in the community alone, without intention to test or refine the method, nor use it as part of evaluation. Three articles described using systems thinking to test and refine the method and the remaining article used systems thinking as an evaluation technique. Twelve articles described using CBSD as part of a wider intervention, with 20 articles describing stand alone interventions. In two articles it was unclear whether the CBSD approach was stand alone or part of a wider intervention.
Eighteen articles did not describe the composition of the facilitation team, eight articles identified that facilitation team members included a mix of academics and community leaders or professionals, seven stated facilitation teams comprised of academic researchers, and one identified a consultancy led facilitation.
Authors used terminology other than CBSD to describe their method of community or stakeholder engagement and qualitative model development (Table 3). Thirteen articles (13 interventions) explicitly describe using CBSD (20, 23, 27, 31, 33, 35, 36, 38, 40, 41, 46, 50, 51). The remaining 21 articles (20 interventions) use other descriptions to explain the methods they use. Five articles (five interventions) describe using GMB with the community to build a causal loop diagram (22, 24, 37, 42, 43), and three articles (three interventions) describe using GMB with the community (25, 29, 49). Three articles (three interventions) describe building a causal loop diagram with the community (26, 30, 48), two articles (two interventions) describe using system dynamics (SD) with the community (28, 44) and two articles (two interventions) describe using participatory GMB (21, 39).
Table 3. Summary of articles and interventions.
Other methods described include; participatory system dynamics modeling (SDM) with the community (34), systems frameworks including CLD’s (32), Stakeholder-Driven Community Diffusion (SDCD) using GMB (52), GMB online with community (45), a participatory action approach of dynamic system mapping and systemic strategy design (47), and a systems-based community-based intervention (CBI) approach, to build a CLD (19).
3.5. Measuring success
All articles reported on more than one outcome (Table 4), identifying 14 themes, which included measuring contribution, engagement and collaborative experience, cultural appropriateness, the implementation process, ownership, trust and relationships, implementation of action, ongoing community engagement and community voice, and unintended consequences. Changes in individual thinking, insights, ideas or mental models, organizational commitment, the system or social norms, data collection, data sources or measurement of change, individual health outcomes, prevention practice, and support for action were also measured.
Table 4. Measures of success by author for 16 articles that measure success.
Of the 34 articles, 18 describe the success of their intervention through subjective author observation and reflection (20, 23, 24, 27, 33–43, 46, 47, 50), that is, where authors describe the success of the intervention, in the absence of additional data collection. The remaining 16 articles use a range of other methods to measure success or effectiveness of the intervention.
The 18 articles that describe their success through author observation and reflection include four named interventions across four articles, WHOSTOPS (20) HFW (24), Portland, a WHOSTOPS pilot community (23) and Nourishing Hawke’s Bay: He wairua tō te kai (27). Fourteen articles did not identify a named intervention.
The remaining 16 articles measured success or effectiveness of their approach using semi-structured interviews at different timepoints (22, 29–31, 48, 51, 52), surveys or questionnaires at different timepoints (19, 29, 45, 48, 49, 52), informal qualitative feedback at different timepoints (25, 44, 49), project documentation (for example meeting minutes) (26, 30), action tracking (action register or on CLD) (21, 26), health measures and population health or education datasets (19), comparative systems thinking case studies (28), stakeholder engagement database (26) and structured conversations and in-depth interviews (32) (Table 5).
Table 5. Evaluation method by author for 16 articles that measure success.
3.6. What influences success
There were numerous implementation factors influencing findings from across the studies. Nineteen themes emerged during the analysis of implementation factors. Of the articles describing success through author observation, the following themes describing implementation factors influenced success: the development of CLD’s; the overall process (including workshops or interviews, CBSD, GMB and other methods); the time allocated to the process (for example, categories included the length of a workshop or the length of the process overall) alongside the timing of different parts of the process (for example, categories included the time taken between workshops, or the time allowed for participants and facilitators to adapt to momentum); the participants who were engaged, methods of engagement and ongoing commitment; and the composition, skills and experience of the facilitation team (Table 6). While one article (20) touched on these issues, it included a focus on building capacity of the use of CBSD in prevention more broadly. As such, this paper identified additional factors as important to outcomes of the approach: supporting a strong process, including utilization of existing structures and ensuring strong collaborative relationships between practice and academia; and using a capacity building approach.
Table 6. Key implementation factors and how they affected findings in 16 studies that measured success.
Descriptions of the effects of implementation factors varied across articles, for example, where time and timing influenced success, one article (48) stated participants:
“found the diagramming activities acceptable, but indicated they needed more time because they were only beginning to understand the concepts when the session ended.”
Another article (26) identified:
“Actions operated on differing timescales, for many there was some delay between the initial planning and the implementation and following there was often adaptation of the action.”
While a third article (32) stated:
“Every group and every individual interviewed emphasized that systems change in communities takes more time than people are accustomed to.”
Additional implementation factors that influenced success in the 16 articles that used non-observation methods were; the role of coordination teams with the opportunity to shape the approach; the ability for a group to come together to implement collective action; providing opportunities for participant feedback on the process; leveraging workshop outputs; the use of systems thinking methods; the strength and quality of relationships and collaboration; the opportunity to combine multiple approaches simultaneously; and a flexible delivery model (accommodating for differences in language, number and timing of workshops, literacy, numeracy, computer literacy or confidence using technology) (Table 6).
Findings from the 16 articles are presented alongside implementation factors in Table 6 with 23 themes identified. Findings included increased community action, increases in strategic thinking, future planning and evaluation, increasing community buy-in and community voice, developing shared visions and goals and creating new, ongoing collaborations. Findings also included building momentum, increasing community contribution and leadership, acknowledging the time required to develop new partnerships and collective thinking, increased understanding of feedback and how it contributes to understanding problems and corresponding action, among many others.
4. Discussion
4.1. Summary of main findings
Measures and concepts of success varied across the articles reviewed, often comprising subjective observations and reflections of study authors, or resulting from semi-structured interviews with key stakeholders. Typical measures of success included community action, collaboration, changes in mental models, or cultural appropriateness. It is difficult to determine specific effects of each implementation factor theme as this was reported to vary across studies (for example, where one had a reported effect in one study, the same factor may have had the opposite effect in another study), which was also reportedly influenced by the characteristics of participants and the context of each approach. We found variation in methods to achieve the same outcome and variation in outcomes sought using the same method describing an emerging field trialing multiple alternate approaches.
Our review builds on reviews by Carey (1), Rusoja (2), and Wilkinson (54) who aimed to investigate the use of systems science or systems thinking in public health or health generally. All three reviews identified various systems methods and terms, with modeling, specifically causal loop diagrams, featuring as one of the most common systems methods used in health. Wilkinson found that although there were many calls to use systems thinking methods, there were few published examples. Our review shows growth in the application of systems thinking, specifically CBSD, and while example evaluations are limited, they are also beginning to grow. Calls from Carey (1) Rouwette (55) and Scott (11) highlight the important stretch beyond model creation as an outcome, to draw attention to research that examines the quality and effectiveness of systems science as a method, with Rouwette specifically calling for examination of successful and unsuccessful efforts (in GMB).
Our review supports findings in a review by Bagnall, Radley (56) that identifies barriers and enablers to implementation of whole systems approaches (WSA’s), noting that leadership, engagement, paying attention to partnerships and building trust (and allowing the time required to do so), governance and shared values, and developing collaborative teams all influenced success. Building on Bagnall’s review, Jayasinghe, Soward (57) found that WSA’s need to include as many domains of capacity building as possible. Domains identified the importance of leadership, in all its forms, alongside partnerships, community engagement and mobilization of resources, all of which were echoed as important implementation factors for success of CBSD approaches in our review. Cilenti, Issel (3) and Littlejohns, Hill (58) have also conducted helpful reviews that explore how system dynamics and CLD’s can be used by communities to realize community action. They found, as in our review, there are times when success may be defined in ways other than community action. This should be considered in future systematic reviews of CBSD and other community-based systems thinking methods.
A recent review (59) developed a framework for measuring success of participatory modeling in systems approaches, and though the study was not focused on community intervention alone, clear similarities were observed between the findings of the current study. Specifically they consider four categories (feasibility, value, change and action, and sustainability) to guide evaluation design of participatory modeling programs. Further parallels exist outside the use of system science, and the findings of this review are supported by other reviews of community capacity and readiness, for example Nagorcka-Smith, Bolton (60) reviewed 26 studies and found shared decision making, resourcing, leadership and facilitation were critical to successful implementation of community initiatives that involved community coalitions. This is echoed by Brush, Mentz (61) who identified the strength of relationships, characteristics and composition of the group influenced success.
4.2. Strengths and limitations
A limitation of this review was the wide variability of terms within systems thinking and the ability to identify those explicitly intending to empower and mobilize a community (as identified in descriptions by Hovmand (9) and Király and Miskolczi (10)). For this reason, the search terms ‘community’ and ‘stakeholder’ were applied to narrow the field of articles down to those most relevant in the community setting, although this may prove overly simplistic without the inclusion of alternatives (for example, using the term ‘participant’). In addition, systems thinking does not include other participatory modeling techniques such as ‘concept mapping’ or ‘group concept mapping’ (62) which were excluded from this review. Other systems methods, such as social network analysis (63), and agent-based modeling (64), were also excluded.
This review did not explore the grey literature as it was anticipated there would be far less empirical measurement of outcomes meaning several examples of CBSD may have been excluded from this review. The review is also limited to articles in the English language. Of 34 papers, few set out to actively evaluate their approach and only 16 had data extracted that could provide insights into implementation factors and measures of success. While some articles collected data at multiple timepoints, others did not, which proved challenging during analysis.
Strengths of this review include the systematic review of all published literature, with all titles, abstracts and full text screened by at least two authors. Analysis was conducted in line with the recommended process for narrative synthesis by Popay and Roberts (17), and in line with PRISMA guidelines (15). An additional strength was the breadth of terms used, which ensured inclusion of findings of other studies that may not have explicitly identified CBSD in their methodology. Adaptation of methods and scripts to apply CBSD is critical to the success of the practice (9). This review captures variations in the approach and compares these against adaptations to measure success based on the direction of the intervention.
4.3. Implications for policy and practice
Despite limited evidence for measurement of success CBSD, our systematic review shows that success is influenced by early design decisions, for example, identifying key stakeholders, composition of the facilitation team/s, timing of workshops, and perceived influence of the process by community members. This indicates it is not only crucial to consider and plan for measurement of success early in the application of CBSD, but that this planning will directly influence implementation, potentially influencing longer term outcomes for the community involved in the approach.
While this review focuses on the use of CBSD in prevention, findings may also be helpful to inform design, implementation and evaluation of other methods that aim to empower community to address complex problems, specifically measures of success, data collection methods, and implementation factors that influence success.
Summarizing the findings from this review for practice, we have developed the following considerations, with each informing the next:
•Consider why you are using CBSD with the community, and be clear on its purpose. What are you aiming to do? What is your end point?
•Identify what change to measure. What will change as a result of you using this approach? Is it community action, individual thinking, change in policy, approach to planning, increased collaboration, increased engagement, or another measure?
•Identify the most appropriate data collection tools. What data collection tools will you use to measure the change you are aiming for? If using surveys, are validated or tested surveys or questionnaires available? If the tool used is novel, is it described in enough detail that it can be replicated by others?
•Consider strategies that will support empowerment of the community after diagram development is complete. How will you care for participants and their contribution after diagram development is complete?
4.4. Future research
In the absence of extensive literature that measures success in CBSD, future research may benefit in looking to studies from other disciplines that have measured effectiveness of group model building (for example, organizational management and organizational change (65)) to draw on other existing and tested frameworks and tools. Examples include Rouwette’s previously developed questionnaire designed to measure communication quality, consensus and commitment to conclusions for those studies aiming to measure insight or collaboration (66), or Fokkinga’s mental model survey (67) for those aiming to measure changes in individual thinking. Measurement frameworks and tools required will vary and will be shaped by the original purpose and setting of the work.
Future attempts to measure success of CBSD should ensure insights are collected from community members and stakeholders at multiple timepoints, including at delayed timepoints after workshops or a CLD has been first developed. This will help determine how community members have applied new insights from the process, and help identify changes in actions, policy or planning at the local level. Evaluation frameworks applied in other participatory modeling approaches may also be helpful. Our review echoes calls from Lee, Hickie (59), emphasizing that evaluation is planned and budgeted for, extending beyond the development of the diagram or CLD. In the case of CBSD, it is important studies plan to capture impacts that stretch beyond modeling and explore impacts within the community.
5. Conclusion
Greater emphasis on measurement of success of future CBSD approaches is required. The use of CBSD and other community-based systems approaches in prevention is growing rapidly. Continuing to synthesize and apply evidence from this research as it emerges will be critical to improve population health. Research teams, alongside the communities they are working with, must articulate what they are seeking to change by using CBSD. Defining success in this way will help identify what will be measured, how it may be measured (including identifying appropriate data collection methods and tools) and will provide a clearer understanding of success.
There are helpful attempts to measure success and effectiveness of CBSD approaches in the published literature. These examples show the importance of design, facilitation strengths and ongoing community engagement as key factors in implementation.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
TF completed the original search, contributed to article screening, full text assessment, data extraction, analysis, preparing initial and ongoing drafts of manuscript, and editing final manuscript. KB contributed to research design, draft manuscripts, and editing final manuscript. EH contributed to draft manuscripts and editing final manuscript. AB and PN-S contributed to article screening and assessment of full texts, draft manuscripts and final manuscript. SA contributed to final research design, provided ongoing supervision, draft manuscripts and editing final manuscript. All authors contributed to the article and approved the submitted version.
Funding
TF holds a research role funded by VicHealth (Victorian Health Promotion Foundation). KB is supported by Heart Foundation Future Leader Fellowship (102047) from the National Heart Foundation of Australia. SA receives/has received funding from the Australian National Health and Medical Research Council (GNTs PPRG 2015440 Promoting CHANGE, GNT1151572 RESPOND, GNT1114118 WHO STOPS, GNT2013563,GNT10458360, GNT2011209, and GNT2002234 PRECIS) the Australian Medical Research Future Fund (DELIVER), VicHealth, Western Alliance, Barwon Health, the European Union Horizon 2020 H2020-SFS-2016-2017 (Co-Create), Novo Nordisk, UK Medical Research Council, Cancer Council Victoria, Deakin University, Melbourne Lord Mayor’s Charitable Foundation, The Australian Prevention Partnership Centre.
Acknowledgments
We would like to acknowledge the ongoing support provided by Deakin University Librarian Liaisons, Health, particularly Lisa Grbin and Frances Beard.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2023.1103834/full#supplementary-material
Abbreviations
CBSD, Community based system dynamics; GMB, Group model building; CLD, Causal loop diagram.
References
1.Carey, G, Malbon, E, Carey, N, Joyce, A, Crammond, B, and Carey, A. Systems science and systems thinking for public health: a systematic review of the field. BMJ Open. (2015) 5:e009002. doi: 10.1136/bmjopen-2015-009002
2.Rusoja, E, Haynie, D, Sievers, J, Mustafee, N, Nelson, F, Reynolds, M, et al. Thinking about complexity in health: a systematic review of the key systems thinking and complexity ideas in health. J Eval Clin Pract. (2018) 24:600–6. doi: 10.1111/jep.12856
3.Cilenti, D, Issel, M, Wells, R, Link, S, and Lich, KH. System dynamics approaches and collective action for community health: an integrative review. Am J Community Psychol. (2019) 63:527–45. doi: 10.1002/ajcp.12305
4.Baugh Littlejohns, L, Baum, F, Freeman, T, and Lawless, A. The value of a causal loop diagram in exploring the complex interplay of factors that influence health promotion in a multisectoral health system in Australia. Health Res Policy Syst. (2018) 16:1–12. doi: 10.1186/s12961-018-0394-x
5.Ison, RL. Systems thinking and practice for action research. The Sage handbook of action research participative inquiry and practice. Sage Publications (2008), 139–158.
7.Checkland, PB. Soft systems methodology. Hum Syst Manag. (1989) 8:273–89. doi: 10.3233/HSM-1989-8405
8.Ulrich, W. Critical heuristics of social planning: a new approach to practical philosophy. Wiley & Sons (1983).
10.Király, G, and Miskolczi, P. Dynamics of participation: system dynamics and participation—an empirical review. Syst Res Behav Sci. (2019) 36:199–210. doi: 10.1002/sres.2580
11.Scott, RJ, Cavana, RY, and Cameron, D. Recent evidence on the effectiveness of group model building. Eur J Oper Res. (2016) 249:908–18. doi: 10.1016/j.ejor.2015.06.078
12.Jackson, MC. Critical systems thinking and the management of complexity. Hoboken, NJ: John Wiley & Sons (2019).
13.World Health Organization. Declaration of Alma-Ata. København: World Health Organization Regional Office for Europe (1978).
14.World Health Organization. Ottawa charter for health promotion, 1986. København: World Health Organization Regional Office for Europe (1986).
15.Page, MJ, McKenzie, JE, Bossuyt, PM, Boutron, I, Hoffmann, TC, Mulrow, CD, et al. The Prisma 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. (2021) 10:1–11. doi: 10.1016/j.jclinepi.2021.03.001
16.Critical Appraisal Skills Programme (CASP). (n.d.). Casp checklists UK: OAP Ltd. Available at: https://casp-uk.net/casp-tools-checklists/
17.Popay, J, Roberts, H, Sowden, A, Petticrew, M, Arai, L, Rodgers, M, et al. (2006). Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version.
18.Braun, V, and Clarke, V. Using thematic analysis in psychology. Qual Res Psychol. (2006) 3:77–101. doi: 10.1191/1478088706qp063oa
19.Jacobs, J, Strugnell, C, Allender, S, Orellana, L, Backholer, K, Bolton, KA, et al. The impact of a community-based intervention on weight, weight-related behaviours and health-related quality of life in primary school children in Victoria, Australia, according to socio-economic position. BMC Public Health. (2021) 21:1–12. doi: 10.1186/s12889-021-12150-4
20.Allender, S, Brown, AD, Bolton, KA, Fraser, P, Lowe, J, and Hovmand, P. Translating systems thinking into practice for community action on childhood obesity. Obes Rev. (2019) 20:179–84. doi: 10.1111/obr.12865
21.Bolton, KA, Fraser, P, Lowe, J, Moodie, M, Bell, C, Strugnell, C, et al. Generating change through collective impact and systems science for childhood obesity prevention: the Genr8 change case study. PLoS One. (2022) 17:e0266654. doi: 10.1371/journal.pone.0266654
22.Jenkins, E, Lowe, J, Allender, S, and Bolton, KA. Process evaluation of a whole-of-community systems approach to address childhood obesity in Western Victoria, Australia. BMC Public Health. (2020) 20:450. doi: 10.1186/s12889-020-08576-x
23.Brown, AD, Bolton, KA, Clarke, B, Fraser, P, Lowe, J, Kays, J, et al. System dynamics modelling to engage community stakeholders in addressing water and sugar sweetened beverage consumption. Int J Behav Nutr Phys Act. (2022) 19:1–10. doi: 10.1186/s12966-022-01363-4
24.Gerritsen, S, Renker-Darby, A, Harré, S, Rees, D, Raroa, DA, Eickstaedt, M, et al. Improving low fruit and vegetable intake in children: findings from a system dynamics, community group model building study. PLoS One. (2019) 14:e0221107. doi: 10.1371/journal.pone.0221107
25.Gerritsen, S, Harré, S, Rees, D, Renker-Darby, A, Bartos, AE, Waterlander, WE, et al. Community group model building as a method for engaging participants and Mobilising action in public health. Int J Environ Res Public Health. (2020) 17:3457. doi: 10.3390/ijerph17103457
26.Maitland, N, Wardle, K, Whelan, J, Jalaludin, B, Creighton, D, Johnstone, M, et al. Tracking implementation within a community-led whole of system approach to address childhood overweight and obesity in south West Sydney, Australia. BMC Public Health. (2021) 21:1233. doi: 10.1186/s12889-021-11288-5
27.McKelvie-Sebileau, P, Rees, D, Tipene-Leach, D, D’Souza, E, Swinburn, B, and Gerritsen, S. Community co-Design of Regional Actions for Children’s nutritional health combining indigenous knowledge and systems thinking. Int J Environ Res Public Health. (2022) 19:4936. doi: 10.3390/ijerph19094936
28.Burke, J, Lich, K, Neal, J, Meissner, H, Yonas, M, and Mabry, P. Enhancing dissemination and implementation research using systems science methods. Int J Behav Med. (2015) 22:283–91. doi: 10.1007/s12529-014-9417-3
29.Calancie, L, Fullerton, K, Appel, JM, Korn, AR, Hennessy, E, Hovmand, P, et al. Implementing group model building with the shape up under 5 community committee working to prevent early childhood obesity in Somerville, Massachusetts. J Public Health Manag Pract. (2020) 28:E43–55. doi: 10.1097/PHH.0000000000001213
30.Cavill, N, Richardson, D, Faghy, M, Bussell, C, and Rutter, H. Using system mapping to help plan and Implement City-wide action to promote physical activity. J Public Health Res. (2020) 9:1759. doi: 10.4081/jphr.2020.1759
31.Morais, LMO, Kuhlberg, J, Ballard, E, Indvik, K, Rocha, SC, Sales, DM, et al. Promoting knowledge to policy translation for urban health using community-based system dynamics in Brazil. Health Res Policy Syst. (2021) 19:53. doi: 10.1186/s12961-020-00663-0
32.Zurcher, KA, Jensen, J, and Mansfield, A. Using a systems approach to achieve impact and sustain results. Health Promot Pract. (2018) 19:15S–23S.
33.Kumar, P, Chalise, N, and Yadama, GN. Dynamics of sustained use and abandonment of clean cooking systems: study protocol for community-based system dynamics modeling. Int J Equity Health. (2016) 15:70. doi: 10.1186/s12939-016-0356-2
34.Macmillan, A, Davies, M, Shrubsole, C, Luxford, N, May, N, Lai Fong, C, et al. Integrated decision-making about housing, energy and wellbeing: a qualitative system dynamics model. Environ Health. (2016) 15:23–34. doi: 10.1186/s12940-016-0098-z
35.Marçal, KE, Fowler, PJ, Hovmand, PS, and Cohen, J. Understanding mechanisms driving family homeless shelter use and child mental health. J Soc Serv Res. (2021) 47:473–85. doi: 10.1080/01488376.2020.1831681
36.Mui, Y, Ballard, E, Lopatin, E, Thornton, RLJ, Pollack Porter, KM, and Gittelsohn, J. A community-based system dynamics approach suggests solutions for improving healthy food access in a low-income urban environment. PLoS One. (2019) 14:e0216985. doi: 10.1371/journal.pone.0216985
37.Naumann, RB, Kuhlberg, J, Sandt, L, Heiny, S, Kumfer, W, Marshall, SW, et al. Integrating complex systems science into road safety research and practice, part 2: applying systems tools to the problem of increasing pedestrian death rates. Injury Prev. (2020) 26:424–31. doi: 10.1136/injuryprev-2019-043316
38.Noubani, A, Diaconu, K, Ghandour, L, El Koussa, M, Loffreda, G, and Saleh, S. A community-based system dynamics approach for understanding factors affecting mental health and health seeking behaviors in Beirut and Beqaa regions of Lebanon. Glob Health. (2020) 16:28. doi: 10.1186/s12992-020-00556-5
39.Noubani, A, Diaconu, K, Loffreda, G, and Saleh, S. Readiness to Deliver person-focused Care in a Fragile Situation: the case of mental health Services in Lebanon. Int J Ment Heal Syst. (2021) 15:21. doi: 10.1186/s13033-021-00446-2
40.Swierad, E, Huang, TTK, Ballard, E, Flórez, K, and Li, S. Developing a Socioculturally nuanced systems model of childhood obesity in Manhattan's Chinese American community via group model building. J Obes. (2020) 2020:4819143. doi: 10.1155/2020/4819143
41.Trani, JF, Ballard, E, Parul, B, and Hovmand, P. Community based system dynamic as an approach for understanding and acting on messy problems: a case study for global mental health intervention in Afghanistan. Confl Heal. (2016) 10
42.Waqa, G, Moodie, M, Snowdon, W, Latu, C, Coriakula, J, Allender, S, et al. Exploring the dynamics of food-related policymaking processes and evidence use in Fiji using systems thinking. Health Res Policy Syst. (2017) 15:74. doi: 10.1186/s12961-017-0240-6
43.Zablith, N, Diaconu, K, Naja, F, El Koussa, M, Loffreda, G, Bou-Orm, I, et al. Dynamics of non-communicable disease prevention, diagnosis and control in Lebanon, a fragile setting. Confl Heal. (2021) 15:4. doi: 10.1186/s13031-020-00337-2
44.Loyo, HK, Batcher, C, Wile, K, Huang, P, Orenstein, D, and Milstein, B. From model to action: using a system dynamics model of chronic disease risks to align community action. Health Promot Pract. (2013) 14:53–61. doi: 10.1177/1524839910390305
45.Chavez-Ugalde, Y, Toumpakari, Z, White, M, De Vocht, F, and Jago, R. Using group model building to frame the commercial determinants of dietary behaviour in adolescence–proposed methods for online system mapping workshops. BMC Med Res Methodol. (2022) 22:1–16. doi: 10.1186/s12874-022-01576-y
46.Deutsch, AR, Lustfield, R, and Jalali, MS. Community-based system dynamics modelling of stigmatized public health issues: increasing diverse representation of individuals with personal experiences. Syst Res Behav Sci. (2022) 39:734–49. doi: 10.1002/sres.2807
47.Egbuonye, NC, Sesterhenn, L, Goldman, B, Pikora, J, and Parmater, J. Bridging the gap between transformative practices and traditional public health approaches. J Public Health Manag Pract. (2022) 28:S151–8. doi: 10.1097/PHH.0000000000001502
48.Frerichs, L, Hassmiller Lich, K, Young, TL, Dave, G, Stith, D, and Corbie-Smith, G. Development of a systems science curriculum to engage rural African American teens in understanding and addressing childhood obesity prevention. Health Educ Behav. (2018) 45:423–34. doi: 10.1177/1090198117726570
49.Frerichs, L, Lich, KH, Funchess, M, Burrell, M, Cerulli, C, Bedell, P, et al. Applying critical race theory to group model building methods to address community violence. Prog Community Health Partnersh. (2016) 10:443–59. doi: 10.1353/cpr.2016.0051
50.Haroz, EE, Fine, SL, Lee, C, Wang, Q, Hudhud, M, and Igusa, T. Planning for suicide prevention in Thai refugee camps: using community-based system dynamics modeling. Asian Am J Psychol. (2021) 12:193–203. doi: 10.1037/aap0000190
51.Browne, J, Walker, T, Brown, A, Sherriff, S, Christidis, R, Egan, M, et al. Systems thinking for aboriginal health: understanding the value and acceptability of group model building approaches. SSM Popul Health. (2021) 15:100874. doi: 10.1016/j.ssmph.2021.100874
52.Calancie, L, Nappi, D, Appel, J, Hennessy, E, Korn, AR, Mitchell, J, et al. Implementing and evaluating a stakeholder-driven community diffusion–informed early childhood intervention to prevent obesity, Cuyahoga County, Ohio, 2018–2020. Prev Chronic Dis. (2022) 19:E03.
53.World Bank. (2021). Country classifications by income. Available at: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups
54.Wilkinson, J, Goff, M, Rusoja, E, Hanson, C, and Swanson, RC. The application of systems thinking concepts, methods, and tools to Global Health practices: an analysis of case studies. J Eval Clin Pract. (2018) 24:607–18. doi: 10.1111/jep.12842
55.Rouwette, EA, Vennix, JA, and Tv, M. Group model building effectiveness: a review of assessment studies. Syst Dyn Rev. (2002) 18:5–45. doi: 10.1002/sdr.229
56.Bagnall, AM, Radley, D, Jones, R, Gately, P, Nobles, J, Dijk, MV, et al. Whole systems approaches to obesity and other complex public health challenges: a systematic review. BMC Public Health. (2019) 19
57.Jayasinghe, S, Soward, R, Dalton, L, Holloway, TP, Murray, S, Patterson, KA, et al. Domains of capacity building in whole-systems approaches to prevent obesity—a “systematized” review. Int J Environ Res Public Health. (2022) 19:10997. doi: 10.3390/ijerph191710997
58.Littlejohns, LB, Hill, C, and Neudorf, C. Diverse approaches to creating and using causal loop diagrams in public Health Research: recommendations from a scoping review. Public Health Rev. (2021) 42:1–13. doi: 10.3389/phrs.2021.1604352
59.Lee, GY, Hickie, IB, Occhipinti, J-A, Song, YJC, Skinner, A, Camacho, S, et al. Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: a scoping review. PLoS One. (2022) 17:1–27. doi: 10.1371/journal.pone.0266125
60.Nagorcka-Smith, P, Bolton, KA, Dam, J, Nichols, M, Alston, L, Johnstone, M, et al. The impact of coalition characteristics on outcomes in community-based initiatives targeting the social determinants of health: a systematic review. BMC Public Health. (2022) 22:1–26. doi: 10.1186/s12889-022-13678-9
61.Brush, BL, Mentz, G, Jensen, M, Jacobs, B, Saylor, KM, Rowe, Z, et al. Success in long-standing community-based participatory research (Cbpr) partnerships: a scoping literature review. Health Educ Behav. (2020) 47:556–68. doi: 10.1177/1090198119882989
62.Kane, M, and Trochim, WM. (2007). Concept mapping for planning and evaluation: Sage publications.
64.Crooks, AT, and Heppenstall, AJ. Introduction to agent-based modelling In: AJ Heppenstall, AT Crooks, LM See, and M Batty, editors. Agent-based models of geographical systems. Dordrecht, Springer Netherlands: (2012). 85–105.
65.Scott, RJ, Cavana, RY, Cameron, D, and Maani, KE (Eds.). (2012). “Evaluation of group model building in a strategy implementation context: a New Zealand government case study.” in The 30th International Conference of the System Dynamics Society.
66.Rouwette, EA. Facilitated modelling in strategy development: measuring the impact on communication, consensus and commitment. J Oper Res Soc. (2011) 62:879–87. doi: 10.1057/jors.2010.78
Keywords: systems thinking, community-based system dynamics, systematic review, success, evaluation, prevention
Citation: Felmingham T, Backholer K, Hoban E, Brown AD, Nagorcka-Smith P and Allender S (2023) Success of community-based system dynamics in prevention interventions: A systematic review of the literature. Front. Public Health. 11:1103834. doi: 10.3389/fpubh.2023.1103834
Edited by:
Duncan Radley, Leeds Beckett University, United KingdomReviewed by:
Jennifer Hall, Bradford Teaching Hospitals NHS Foundation Trust, United KingdomJames Nobles, Leeds Beckett University, United Kingdom
Copyright © 2023 Felmingham, Backholer, Hoban, Brown, Nagorcka-Smith and Allender. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Tiana Felmingham, dGZlbG1pbmdoYW1AZGVha2luLmVkdS5hdQ==