Engaging Healthcare Staff and Stakeholders in Healthcare Simulation Modeling to Better Translate Research Into Health Impact: A Systematic Review

Objective: To identify processes to engage stakeholders in healthcare Simulation Modeling (SM), and the impacts of this engagement on model design, model implementation, and stakeholder participants. To investigate how engagement process may lead to specific impacts.

Data Sources: English-language articles on health SM engaging stakeholders in the MEDLINE, EMBASE, Scopus, Web of Science and Business Source Complete databases published from inception to February 2020.

Study Design: A systematic review of the literature based on a priori protocol and reported according to PRISMA guidelines.

Extraction Methods: Eligible articles were SM studies with a health outcome which engaged stakeholders in model design. Data were extracted using a data extraction form adapted to be specific for stakeholder engagement in SM studies. Data were analyzed using summary statistics, deductive and inductive content analysis, and narrative synthesis.

Principal Findings: Thirty-two articles met inclusion criteria. Processes used to engage stakeholders in healthcare SM are heterogenous and often based on intuition rather than clear methodological frameworks. These processes most commonly involve stakeholders across multiple SM stages via discussion/dialogue, interviews, workshops and meetings. Key reported impacts of stakeholder engagement included improved model quality/accuracy, implementation, and stakeholder decision-making. However, for all but four studies, these reports represented author perceptions rather than formal evaluations incorporating stakeholder perspectives. Possible process enablers of impact included the use of models as “boundary objects” and structured facilitation via storytelling to promote effective communication and mutual understanding between stakeholders and modelers.

Conclusions: There is a large gap in the current literature of formal evaluation of SM stakeholder engagement, and a lack of consensus about the processes required for effective SM stakeholder engagement. The adoption and clear reporting of structured engagement and process evaluation methodologies/frameworks are required to advance the field and produce evidence of impact.

What is Known on This Topic

• Simulation modeling is an effective research methodology to address complex and “wicked” problems in healthcare and public health.

• Involving stakeholders in healthcare simulation modeling is assumed to produce better (and more relevant) models which are more readily accepted by problem owners and thereby more likely to be implemented, but there is limited evidence to guide choices of methods for engaging stakeholders to enhance the design and implementation of these models.

What This Study Adds

• We document the large gap in the current literature of formal evaluation of SM stakeholder engagement, and a lack of consensus about the processes required for effective SM stakeholder engagement.

• Processes used to engage stakeholders in healthcare simulation modeling are heterogenous and often ill-defined in the literature, generally involving multiple stakeholder types across multiple simulation modeling stages.

• Possible process enablers of impact are the use of models as “boundary objects” and structured facilitation via storytelling for non-technical communication of model logic. These enablers may work by providing a common language and mutual understanding between stakeholders and modelers.

• Adoption and clear reporting of structured engagement and process evaluation methodologies/frameworks are required to advance the field.

Introduction

Healthcare decision-makers are facing increasingly “wicked” problems, which have both a technical (complex and uncertain symptoms and solutions) and social (divergent stakeholder perspectives) dimensions (1, 2). Confronting the technical dimension requires research methods which can account for scientific complexity and uncertainty whilst addressing the social dimension requires stakeholders to be engaged in the research process in order to produce solutions with real-world utility (3, 4).

Simulation Modeling (SM) is an established but historically under-utilized methodology in healthcare (5). SM creates a virtual environment which captures dynamic, interdependent and emergent system behaviors in formal models or mathematical representations (5, 6). These models can be used to “advance the understanding of the system or process, communicate findings, and inform management and policy design” (6). SM comprises three methods—system dynamics, discrete event simulation, and agent-based modeling—which Marshall et al. claim are “well suited to healthcare delivery problems” (6).

SM has increasingly been combined with approaches intended to engage stakeholders in the modeling process. Engaging stakeholders, traditionally managers and clinicians in the relevant healthcare field, has been claimed to yield both more technically and socially robust simulation solutions (7) and improve on the poor model implementation that has plagued SM for many years (5, 7–11). Barreteau et al. outline three expected benefits of combining a participatory process with SM: (1) to upgrade the quality of a simulation model, (2) to improve the suitability of the simulation model's use (implementation), and (3) to support participation itself, and account for different perspectives (function of models within participatory process) (12). Despite these expected benefits, SM stakeholder engagement research and practice in healthcare lags behind other sectors (e.g., defense and commerce) (13).

Knowing how best to involve themselves or others in research is a challenge for clinicians, decision-makers and stakeholders in healthcare, as well as for the researchers. Yet involving frontline clinicians, decision-makers and other relevant stakeholders in research that aims to promote a change in practice is key to translating research “off the shelf and into practice” (14). Several authors have identified barriers to successful stakeholder engagement unique to SM research. Jahangirian et al. determined the primary causal factor of poor engagement as the “communication gap between simulation and stakeholder groups” as simulation modelers may have particularly technical backgrounds (9). Brailsford et al., drawing on their experiences within the UK, discuss commonly encountered barriers, including cultural differences and ethical hurdles (8). Whilst an understanding of barriers to engagement is important, guiding decisions in practice about how to effectively engage different stakeholders in designing healthcare simulation models requires further understanding of who should be engaged, when this engagement should occur, and how this engagement should be done to generate the intended impacts. Several simulation studies in healthcare have provided descriptions, reflections or evaluations of their stakeholder engagement process (15, 16), however, there is no coherent body of literature in this area.

The aim of this review is to systematically synthesize the evidence on how far and by what means stakeholder engagement in SM results in outcomes with more practical utility and prospect of successful implementation. A key objective is to identify the processes used to meaningfully engage stakeholders in SM research and to analyze the impacts of this engagement in enhancing the design and implementation of healthcare simulation models. In order to accomplish this, we analyzed the extent to which these intended purposes or expected benefits of SM have materialized in applications to healthcare problems, and we identify the contribution of engagement processes to facilitating this.

Methods

Study Design

This systematic review is reported according to the PRISMA statement (17) and used an a priori established protocol (Supplementary Material).

Eligibility Criteria and Search Strategy

Eligible articles were original studies that (1) used dynamic SM (intervention), (2) reported a health-related primary outcome (context), (3) engaged stakeholders during the model design stage (population), and (4) reported stakeholder engagement impact (outcome). The search was not limited to a specific study design and did not include a comparator. English-language articles were searched in MEDLINE, EMBASE, Scopus, Web of Science and Business Source Complete databases published from inception to February 2020. Common keywords included: simulation, system dynamics, discrete event or agent based; health care, healthcare, hospital, primary care, public health, health policy or health service; group model building, stakeholder, client, customer, implementation, focus group, interview, steering group, advisory board, advisory committee, co-design, co-production; and participatory simulation or participatory model. Full details are in Appendix A including the title and abstract screening criteria and the full text inclusion criteria.

Study Selection, Data Extraction and Analysis

Title and abstract, and then full-text screening (see Appendix A for full details) were conducted by TZ (all studies) and KL (25% of studies), with disagreement resolved by discussion. Data were extracted using a data extraction form adapted from Concannon et al. (18) and located in Appendix A.

Data were analyzed using summary statistics, deductive and inductive content analysis, (19) as well as a narrative synthesis approach (20). Summary statistics were used to analyze study characteristics. Content analysis was applied to synthesize qualitative data describing the participatory process and intended or reported impacts of this process. We used matrices to explore the overlap between process characteristics and intended or reported impacts, in order to map how the nature of the process may link to impacts. To obtain a richer understanding of the participatory processes, a narrative synthesis approach was used to analyze the role of stakeholder engagement activities within the SM process. Topic areas were categorized according to Mielczarek and Uziałko-Mydlikowska (21), and stakeholder types were categorized according to an adaptation of the 7P's framework, with purchasers and payers combined into a single category (14). The generic stages of SM lifecycles (Figure 1) were used to represent modeling stages– problem formulation, conceptual modeling, computer modeling, model verification and validation, experimentation and implementation (22). Other stages which engaged stakeholders that didn't fit into the generic stages were inductively coded. The intended and reported benefits of the participatory process were coded within a framework adapted from Barreteau et al., comprising three broad types of benefits for (1) the design of the model, (2) the implementation of the model, and (3) the stakeholder participants (12). Within this framework, inductive content analysis was used to identify and quantify the sub-groups of benefits.

FIGURE 1

Figure 1. Generic stages in SM.

The evidence synthesis concentrated on authors' reporting of the participatory process in SM studies, which meant that outcome measures from the studies were not included. Therefore, no formal assessment of risk of bias was necessary either in individual studies or across studies (23).

Results

The search yielded 1,682 titles and abstracts for initial screening, with 119 articles included for full-text screening. Of full-text articles screened, 29 met the eligibility criteria, with a further three identified and included from included articles reference lists (see Figure 2 PRISMA diagram). The final 32 articles reported on 27 studies (see Table 1 for a summary of included studies).

FIGURE 2

Figure 2. PRISMA flow diagram of study selection.

TABLE 1

Table 1. Summary of studies included in the systematic review (n = 27).

Study Characteristics

Of the 27 studies included, the majority were conducted in the UK (n = 8, 30%), US (n = 7, 26%) and Australia (n = 4, 15%). The most common topic areas were Health and Care Systems Operation (56%) and Epidemiology, Health Promotion and Disease Prevention (41%), with few studies in Health and Care Systems Design (3%) and Extreme Events Planning (3%). None of the included studies addressed the topic of Medical Decision Making. Years of publication ranged from 2000 to 2019, with 78% published since 2014 (n = 21). There was a change in trend in study topic areas over time: initially dominated by Health and Care Systems Operation, and in recent years by Epidemiology, Health Promotion and Disease Prevention (Figure 3).

FIGURE 3

Figure 3. Evolution of study topic areas over time.

Stakeholder Participants

The number of studies reporting stakeholder engagement during different stages of SM are shown in Figure 4. The type and number of stakeholders involved in the participatory process varied widely between studies. The most frequently-engaged stakeholders were Providers (n = 23), with less engagement with Policy-makers (n = 7), Purchasers/Payers (n = 9) and Patients/Public (n = 6). Each study, on average, engaged two different types of stakeholder participants. Further details on the numbers and types of stakeholders are in Appendix B. Recruitment methods for stakeholders are also in Appendix B.

FIGURE 4

Figure 4. Number of studies reporting stakeholder engagement during generic (blue) and openly coded (green) stages of SM. Common stages of engagement aside from conceptual model building were problem formulation (74%). model verification & validation (74%) and experimentation & implementation (59%). It was relatively uncommon for studies to engage stakeholders during the computer model building stage (11%). however, collecting data or parameters through the engagement of stakeholders was used by almost half the studies as a means of quantifying the computer model (41%). Only two studies (7%) engaged stakeholders in evaluating the participatory process in the simulation study.

The Participatory Process: Stakeholder Engagement Stages, Modes and Activities

The stages during which stakeholders were engaged are depicted graphically in Figure 4. All but one study engaged stakeholders in more than one stage of the SM process. In nine studies (33%), stakeholders were engaged in all the generic stages (from the beginning to the end of the SM lifecycle—excluding computer model building). There were four primary modes employed by studies in the engagement of stakeholders: discussion/dialogue (n = 13, 48%), interviews (n = 11, 41%), workshops (n = 8, 30%), and meetings (n = 7, 26%), with some variation across SM stages. Specifically, interviews were most common earlier in the SM process while workshops were commonly used in the mid and late stages of the process. Workshops, discussion/dialogue and meetings generally involved direct model interaction, where the model acted as a communication vehicle (n = 18, 67%), allowing stakeholders to physically manipulate and “play” with the model design. Some studies provided descriptions about how they facilitated this input, which ranged from structured and active methods where stakeholders were asked specific questions (31) or engaged in purposeful storytelling exercises (32), to unstructured and passive methods where stakeholders provided feedback about or annotated an existing model (30, 40). More structured methods of facilitation were used in early stages when studies were engaging stakeholders in designing the model from scratch (24, 31, 32, 36, 37, 44, 53), and more passive methods were used when stakeholders were engaged at a later stage and a draft model had already been designed (16, 30, 35, 38, 40, 48). Further details about the Modes of Engagement & Facilitation are found in Appendix C.

Assessing the Impacts of Stakeholder Engagement

There were four types of impacts reported from stakeholder engagement on model design: (1) increased relevance of the problem addressed (n = 4, 14%), (2) better quality/accuracy of the model for its purpose (n = 9, 33%), (3) improved identification or access to better data (n = 4, 15%), and (4) expertise from a range of perspectives (n = 9, 31%). Across the SM stages, increased relevance of the problem was most reported during problem formulation, while better quality/accuracy of the model was most reported during the conceptual modeling phase.

There were three types of impacts reported from the participatory process on the stakeholder participants: (1) productive discussion or shared understanding of the problem (n = 8, 28%), (2) “learning” (n = 5, 17%), and (3) better problem solving or decision-making (n = 10, 35%). The fact that models require perspectives and assumptions to be made explicit in a graphical representation which imitates the real system meant that several studies found that interacting with the model created productive discussion between stakeholders, particularly those from different disciplines (24–26, 49). There were also three types of impacts reported from stakeholder engagement on model implementation: (1) refined use of terminology (n = 1, 3%) (36), (2) greater acceptance or ownership of the model (n = 5, 17%), and (3) improved implementation or suitable use of the model (n = 8, 27%).

In the overlap between the participatory process and impacts, studies involving direct model interaction were more likely to report benefits for stakeholder ownership of the model (56 vs 11%) and productive discussion & shared understanding (61 vs. 22%), vs. studies without direct model interaction. Read more about the impacts of participatory process on model design, implementation and participants in Appendix D.

Specific Processes Used to Engage Stakeholders in Simulation Modeling

Some studies provided specific details about the process used to combine stakeholder engagement and SM and how to do this well (26, 29, 32, 33, 49). This can provide practitioners and decision-makers, as well as researchers, with useful guides for engaging in such processes. These processes included: the Collaborative Hybridization Process (29), an adapted dynamic synthesis methodology (49), and the Modeling Approach that is Participatory Iterative for Understanding (MAPIU) (26). Descriptions and examples of these processes are outlined in Appendix E.

Discussion

We have systematically reviewed the ways in which studies using SM have engaged stakeholders through participatory processes. We reviewed these participatory processes on their reported abilities to improve the design and use of models in healthcare as well as produce desired impacts on stakeholder participants. The reported processes used to engage stakeholders in healthcare SM were heterogenous, but there were common characteristics in terms of the stages, modes and activities through which engagement is facilitated. Studies mostly commonly involved provider stakeholders, across multiple SM stages, using discussion/dialogue, interviews, workshops and meetings as key modes of engagement. In addition to conceptual modeling, many studies engaged stakeholders in the adjacent stages of problem formulation and model verification and validation, as well as during the later stages of experimentation and implementation. Interviews were mostly used earlier in the SM process while workshops were mostly used in the mid and late stages of the process. Key reported impacts of stakeholder engagement included improved model quality/accuracy, implementation, and stakeholder decision-making.

The Link Between Stakeholder Engagement Process and Impact

The communication gap between stakeholders and modelers has been identified as a primary causal factor driving consistently poor rates of model implementation in healthcare SM (9, 54). This review has identified two possible mechanisms by which engagement processes, via improved model design and stakeholder impacts, may lead to improved model implementation.

Firstly, we found that direct interaction between stakeholders and the model seemed to influence interpersonal communication (between stakeholders themselves and between stakeholders and modelers), leading to stakeholder impacts of more productive discussion and shared understanding, and implementation impacts of greater ownership and acceptance of the model. This is likely because the model operates as a “boundary object” (55), a visual “multi-interpretable, consistent transparent, and verifiable representation of reality” (56). To effectively use models in this way, Rose et al. recommend: using specific conventions to describe model components and interactions to create a common language; using an early simple model to teach stakeholders the model concepts; and allowing hands-on stakeholder exposure to the model user interface (57). As such, using models as “boundary objects” may provide structures around which to base effective communication, providing visual aids for stakeholders to view the whole problem system and better identify areas for solutions.

More structured methods of stakeholder engagement, i.e., the use of specific questions (31) or purposeful storytelling exercises (32), were also associated with improved quality of models for their purpose and helped to incorporate diverse perspectives and expertise into the model design. Freebairn et al. discussed a particularly effective example of using “storytelling” to facilitate communication of the model structure through clinical case histories of individuals–a thought process familiar to clinical stakeholders (32) During storytelling the modelers are better able to use language that the stakeholders can understand and relate to. Also, having the stories allows stakeholders to give the modelers an increased understanding of the complexity of “wicked” problems and complexities associated with populations affected by these problems. The use of storytelling can exemplify the individual trajectories of agents, communicating the ability of the model to capture the evolution of agents over time. During the process evaluation, participants reported that while the sophisticated and highly technical nature of the model remained a barrier in communicating easy to understand policy messages, the use of storytelling to compliment the model outputs was a “particularly valuable tool” to improve mutual understanding of the model (32). That greater understanding contributes to improving the modeling, confidence in the model and ownership. This example suggests that storytelling may provide a useful addition to the “boundary object” approach, allowing communication of highly technical model elements in a more easily understood way. It's also likely to contribute to successful design, confidence, model ownership and future implementation as stories seem to get the message across people from different disciplines.

Reporting and Evaluation of Stakeholder Engagement in SM in the Literature

Many of the methodological issues faced by this review were due to a lack of standardized or detailed reporting of stakeholder engagement, and insufficient reporting was one of the primary reasons for article exclusion. This lack of reporting comprised the details of the process itself and adequate evaluations of the engagement process. Only five studies provided specific details about the stakeholder engagement process (26, 29, 32, 33, 49), and a mere four studies reported on a process evaluation from the stakeholder perspective (15, 30, 49, 53). For the remaining studies, it was difficult to distinguish between intended and reported impacts that were observed or realized during the process as the reporting was based solely on the authors' reflections.

A recent framework from the environmental model building field provides a possible solution to the lack of standardized reporting of stakeholder engagement that may be equally applicable to healthcare SM. Gray et al. propose a four-dimensional reporting framework (4P) which includes: (1) the Purpose for using a participatory SM approach (i.e., intended impacts), (2) the Process by which stakeholders were involved in model building or evaluation; (3) the Partnerships that were defined and participants that were chosen; and (4) the Products that resulted from these efforts (i.e., actual impacts) (58). A detailed breakdown of each of these dimensions is provided by the authors in addition to 4 exemplar case studies. This could be supported by the adaptation of one of the engagement processes identified in our review. The MAPIU is easily generalizable and provides several frameworks for designing a participatory SM process, including a classification system for stakeholders, and frameworks for how different stakeholder contributions fit into the MAPIU process and what types of communication should be considered (26). Guidance on process evaluation is sparser, with Esmail et al.'s systematic review identifying only two studies reporting quantitative results, with most formal evaluations relying on qualitative, self-reported retrospective accounts of the engagement experience (59). Future research should focus on the development and validation of measures and methods for rigorous evaluation of engagement in healthcare SM which should be an a priori embedded component of the research design.

A limitation of this review is the timeframe, which included studies published until March 2020.The crisis and transformation occurring in health care since February 2020 due to the COVID-19 pandemic is deliberately not captured here. The COVID-19 pandemic has resulted in rapid changes inside health systems including changes in direct care procedures and the adoption of remote care through new technologies, with a corresponding global burden of high health care worker stress. Therefore, health care stakeholder involvement for SM during COVID-19 would involve different, crisis driven approaches, and is the subject of a separate subsequent project.

Conclusion

This review explored the process by which studies engaged stakeholders in healthcare SM and the impacts of the engagement process on model design, model implementation and stakeholders. We found that engagement of stakeholders in the SM process was common during multiple stages, involved informal discussion as well as more formal one-to-one interviews or group workshops, and was facilitated by a range of more or less structured activities for model building, with structured activities associated with improved model quality and ability to capture diverse perspectives and expertise. Key enablers reported by authors and stakeholder participants were the use of the evolving model as a “boundary object” to facilitate communication and storytelling to communicate the model logic, complexities and interactions in a non-technical way. We suggest the adoption and clear reporting of structured engagement and process evaluation methodologies/frameworks to enable high-quality stakeholder involvement, improve SM confidence and ownership by healthcare staff and decision-makers, and ultimately lead to implementation of optimal interventions identified by SM that contribute toward better health care systems.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

TZ, KL, IM, and JE devised the research and developed the research questions in collaboration with DS and JH. Search strategy was devised by TZ with assistance by KL and JE. Articles were screened by TZ and KL and fulltext extraction performed by TZ with technical guidance and assistance by KL and JE. Manuscript was drafted by TZ. All authors provided inputs into subsequent revisions.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

This study is part of a Ph.D. dissertation and was only feasible thanks to the support of a wide team of researchers at Monash University, Eastern Health, and Turning Point.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2021.644831/full#supplementary-material

References

1. Plsek PE, Greenhalgh T. The challenge of complexity in health care. BMJ. (2001) 323:625. doi: 10.1136/bmj.323.7313.625

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Head BW. Wicked problems in public policy. Public Policy. (2008) 3:101.

Google Scholar

3. Selby JV, Beal AC, Frank L. The patient-centered outcomes research institute (PCORI) national priorities for research and initial research agenda. JAMA. (2012) 307:1583–4. doi: 10.1001/jama.2012.500

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Aff. (2010) 29:1834–41. doi: 10.1377/hlthaff.2010.0675

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Eldabi T editor. Implementation issues of modeling healthcare problems: Misconceptions and lessons. Proceedings of the 2009 Winter Simulation Conference (WSC). Austin, TX: IEEE (2009). doi: 10.1109/WSC.2009.5429192

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Marshall DA, Burgos-Liz L, Ijzerman MJ, Osgood ND, Padula WV, Higashi MK, et al. Applying dynamic simulation modeling methods in health care delivery research— The SIMULATE checklist: report of the ISPOR simulation modeling emerging good practices task force. Value in Health. (2015) 18:5–16. doi: 10.1016/j.jval.2014.12.001

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Harper PR, Pitt MA. On the challenges of healthcare modelling and a proposed project life cycle for successful implementation. J Operational Res Society. (2004) 55:657–61. doi: 10.1057/palgrave.jors.2601719

CrossRef Full Text | Google Scholar

8. Brailsford SC, Bolt T, Connell C, Klein JH, Patel B editors. Stakeholder engagement in health care simulation. Proceedings of the 2009 Winter Simulation Conference (WSC). Austin, TX: IEEE (2009). doi: 10.1109/WSC.2009.5429190

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Jahangirian M, Borsci S, Shah SGS, Taylor SJE. Causal factors of low stakeholder engagement: a survey of expert opinions in the context of healthcare simulation projects. Simulation. (2015) 91:511–26. doi: 10.1177/0037549715583150

CrossRef Full Text | Google Scholar

10. Long KM, Meadows GN. Simulation modelling in mental health: A systematic review. Journal of Simulation. (2018) 12:76–85. doi: 10.1057/s41273-017-0062-0

CrossRef Full Text | Google Scholar

11. Fone D, Hollinghurst S, Temple M, Round A, Lester N, Weightman A, et al. Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health. (2003) 25:325–35. doi: 10.1093/pubmed/fdg075

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Barreteau O, Bots P, Daniell K, Etienne M, Perez P, Barnaud C, et al. Participatory Approaches. In: Edmonds B, Meyer R, editors. Simulating Social Complexity: A Handbook. Berlin, Heidelberg: Springer Berlin Heidelberg. (2013). p. 197–234. doi: 10.1007/978-3-540-93813-2_10

CrossRef Full Text | Google Scholar

13. Jahangirian M, Naseer A, Stergioulas L, Young T, Eldabi T, Brailsford S, et al. Simulation in health-care: lessons from other sectors. Operational Res. (2012) 12:45–55. doi: 10.1007/s12351-010-0089-8

CrossRef Full Text | Google Scholar

14. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. (2012) 27:985–91. doi: 10.1007/s11606-012-2037-1

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Freebairn L, Atkinson JA, Kelly PM, McDonnell G, Rychetnik L. Decision makers' experience of participatory dynamic simulation modelling: methods for public health policy. BMC Medical Informatics and Decision Making. (2018) 18:131. doi: 10.1186/s12911-018-0707-6

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Lane D, Monefeldt C, Husemann E, Lane DC. Client involvement in simulation model building: hints and insights from a case study in a London hospital. Health Care Manag Sci. (2003) 6:105–16. doi: 10.1023/A:1023385019514

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. (2009) 6:e1000097. doi: 10.1371/journal.pmed.1000097

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med. (2014) 29:1692–701. doi: 10.1007/s11606-014-2878-x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. (2008) 62:107–15. doi: 10.1111/j.1365-2648.2007.04569.x

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. A Product From the ESRC Methods Programme. Version 1. ESRC (2006).

Google Scholar

21. Mielczarek B, Uziałko-Mydlikowska J. Application of computer simulation modeling in the health care sector: a survey. Simulation. (2010) 88:197–216. doi: 10.1177/0037549710387802

CrossRef Full Text | Google Scholar

22. Yapa ST. A structured approach to rapid simulation model development: Sheffield Hallam University. (2006).

Google Scholar

23. Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. (2014) 14:89. doi: 10.1186/1472-6963-14-89

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Atkinson JA, O'Donnell E, Wiggers J, McDonnell G, Mitchell J, Freebairn L, et al. Dynamic simulation modelling of policy responses to reduce alcohol-related harms: rationale and procedure for a participatory approach. Public Health Research & Practice. (2017) 27:2711707. doi: 10.17061/phrp2711707

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Freebairn L, Rychetnik L, Atkinson JA, Kelly P, McDonnell G, Roberts N, et al. Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling. Health Research Policy and Systems. (2017) 15:83. doi: 10.1186/s12961-017-0245-1

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Baldwin LP, Eldabi T, Paul RJ. Simulation in healthcare management: a soft approach (MAPIU). Simulation Modelling Practice and Theory. (2004) 12:541–57. doi: 10.1016/j.simpat.2004.02.003

CrossRef Full Text

27. Baldwin LP, Eldabi T, Paul RJ, Burroughs AK. Using simulation for the economic evaluation of liver transplantation. Proceedings of the 32nd conference on Winter simulation. (2000) p. 1963–70.

Google Scholar

28. Barbrook-Johnson P, Badham J, Gilbert N. Uses of Agent-Based Modeling for Health Communication: the TELL ME Case Study. Health Commun. (2017) 32:939–44. doi: 10.1080/10410236.2016.1196414

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Bell D, Cordeaux C, Stephenson T, Dawe H, Lacey P, O'Leary L, et al. Designing effective hybridization for whole system modeling and simulation in healthcare. 2016 Winter Simulation Conference, WSC 2016. (2016) 0:1511–22. doi: 10.1109/WSC.2016.7822202

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Bowers J, Ghattas M, Mould G. Success and failure in the simulation of an Accident and Emergency department. J Simulat. (2009) 3:171–8. doi: 10.1057/jos.2009.5

CrossRef Full Text | Google Scholar

31. de Andrade L, Lynch C, Carvalho E, Rodrigues CG, Vissoci JR, Passos GF, et al. System dynamics modeling in the evaluation of delays of care in ST-segment elevation myocardial infarction patients within a tiered health system. PLoS ONE [Electronic Resource]. (2014) 9:e103577. doi: 10.1371/journal.pone.0103577

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Freebairn L, Atkinson JA, Osgood ND, Kelly PM, McDonnell G, Rychetnik L. Turning conceptual systems maps into dynamic simulation models: An Australian case study for diabetes in pregnancy. PLoS ONE. (2019) 14:e0218875. doi: 10.1371/journal.pone.0218875

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Freebairn L, Atkinson JA, Kelly P, McDonnell G, Rychetnik L. Simulation modelling as a tool for knowledge mobilisation in health policy settings: a case study protocol. Health Research Policy and Systems. (2016) 14:71. doi: 10.1186/s12961-016-0143-y

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Giesen E, Ketter W, Zuidwijk R. An agent-based approach to improving resource allocation in the dutch youth health care sector. 17th European Conference on Information Systems, ECIS 2009. (2009).

Google Scholar

35. Glasgow SM, Perkins ZB, Tai NRM, Brohi K, Vasilakis C. Development of a discrete event simulation model for evaluating strategies of red blood cell provision following mass casualty events. Eur J Oper Res. (2018) 270:362–74. doi: 10.1016/j.ejor.2018.03.008

CrossRef Full Text | Google Scholar

36. Hassmiller Lich K, Urban JB, Frerichs L, Dave G. Extending systems thinking in planning and evaluation using group concept mapping and system dynamics to tackle complex problems. Eval Program Plann. (2017) 60:254–64. doi: 10.1016/j.evalprogplan.2016.10.008

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Homa L, Rose J, Hovmand PS, Cherng ST, Riolo RL, Kraus A, et al. A participatory model of the paradox of primary care. Ann Fam Med. (2015) 13:456–65. doi: 10.1370/afm.1841

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Hung GR, Whitehouse SR, Oneill C, Gray AP, Kissoon N. Computer modeling of patient flow in a pediatric emergency department using discrete event simulation. Pediatr Emerg Care. (2007) 23:5–10. doi: 10.1097/PEC.0b013e31802c611e

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Johnson O, Hall P, Hulme C, Johnson OA, Hall PS, NETIMIS. Dynamic simulation of health economics outcomes using big data. Pharmacoeconomics. (2016) 34:107–14. doi: 10.1007/s40273-016-0384-1

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Lattimer V, Brailsford S, Turnbull J, Tarnaras P, Smith H, George S, et al. Reviewing emergency care systems I: insights from system dynamics modelling. Emergency Medicine J. (2004) 21:685–91. doi: 10.1136/emj.2002.003673

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Leskovar R, Accetto R, Baggia A, Lazarevic Z, Vukovic G, Pozun P. Discrete event simulation of administrative and medical processes. Zdravniski Vestnik-Slovenian Medical Journal. (2011) 80:345–53.

Google Scholar

42. Lote R, Williams EJ, Ulgen OM, Otamendi J, Bargiela A, Montes JL, et al. Simulation of medical laboratory operations to achieve optimal resource allocation. ECMS (2009) 249–55. doi: 10.7148/2009-0249-0255

CrossRef Full Text | Google Scholar

43. Mackay M, Qin S, Clissold A, Hakendorf P, Ben-Tovim D, McDonnell G, et al. Patient flow simulation modelling - an approach conducive to multi-disciplinary collaboration towards hospital capacity management. (2013) 50–6.

Google Scholar

44. Matchar DB, Ansah JP, Bayer S, Hovmand P, Amazon S, ExtendSim, et al. Simulation modeling for primary care planning in Singapore. 2016 Winter Simulation Conference, WSC 2016. (2016) 0:2123–34. doi: 10.1109/WSC.2016.7822255

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Mitchell SG, Lounsbury D, Li Z, Schwartz RP, Gryczynski J, Kirk A, et al. Application of system dynamics to inform a model of adolescent SBIRT implementation in primary care settings. Drug and Alcohol Dependence. (2017) e146. doi: 10.1016/j.drugalcdep.2016.08.404

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Murphy GT, MacKenzie A, Guy-Walker J, Walker C. Needs-based human resources for health planning in Jamaica: using simulation modelling to inform policy options for pharmacists in the public sector. Human Resources for Health. (2014) 12. doi: 10.1186/1478-4491-12-67

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Roberts N, Li V, Atkinson JA, Heffernan M, McDonnell G, Prodan A, et al. Can the target set for reducing childhood overweight and obesity be met? A System Dynamics Modelling Study in New South Wales, Australia. Systems Research and Behavioral Science. (2019) 36:36–52. doi: 10.1002/sres.2555

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Rosmulder R, Krabbendam J, Kerkhoff A, Houser C, Luitse J. Computer simulation within action research: a promising combination for improving healthcare delivery? Systemic Practice & Action Research. (2011) 24:397–412. doi: 10.1007/s11213-011-9191-y

CrossRef Full Text | Google Scholar

49. Rwashana AS, Nakubulwa S, Nakakeeto-Kijjambu M, Adam T. Advancing the application of systems thinking in health: understanding the dynamics of neonatal mortality in Uganda. Health Research Policy and Systems. (2014) 12:36. doi: 10.1186/1478-4505-12-36

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Semwanga AR, Nakubulwa S, Adam T. Applying a system dynamics modelling approach to explore policy options for improving neonatal health in Uganda. Health Res Policy Sys. (2016) 14:35. doi: 10.1186/s12961-016-0101-8

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Uebelherr JM, Hondula DM, Johnston EW, Arizona State University CfST Environmental Policy S Arizona State University DTN . Innovative participatory agent based modeling using a complexity governance perspective. 16th Annual International Conference on Digital Government Research, dgo 2015. (2015) 307–8. doi: 10.1145/2757401.2757447

CrossRef Full Text | Google Scholar

52. Uriarte AG, Zúñiga ER, Moris MU, Ng AHC. System design and improvement of an emergency department using Simulation-Based Multi-Objective Optimization. 31st Euro Mini Conference on Improving Healthcare: New Challenges, New Approaches. (2015) 616. doi: 10.1088/1742-6596/616/1/012015

CrossRef Full Text | Google Scholar

53. Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory system dynamics modeling: increasing stakeholder engagement and precision to improve implementation planning in systems. Administration and Policy in Mental Health and Mental Health Services Research. (2016) 43:834–49. doi: 10.1007/s10488-016-0754-1

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Slattery P, Saeri AK, Bragge P. Research co-design in health: a rapid overview of reviews. Health Research Policy and Systems. (2020) 18:17. doi: 10.1186/s12961-020-0528-9

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Leigh Star S. This is Not a Boundary Object: Reflections on the Origin of a Concept. Science, Technology, & Human Values. (2010) 35:601–17. doi: 10.1177/0162243910377624

CrossRef Full Text | Google Scholar

56. van Bruggen A, Nikolic I, Kwakkel J. Modeling with stakeholders for transformative change. Sustainability. (2019) 11:825. doi: 10.3390/su11030825

CrossRef Full Text | Google Scholar

57. Rose J, Homa L, Hovmand P, Kraus A, Burgess K, Biswas A, et al editors. Boundary objects for participatory group model building of agent-based models. 2015 48th Hawaii International Conference on System Sciences. (2015). doi: 10.1109/HICSS.2015.357

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Gray S, Voinov A, Paolisso M, Jordan R, BenDor T, Bommel P, et al. Purpose, processes, partnerships, and products: four Ps to advance participatory socio-environmental modeling. Ecological Applications. (2018) 28:46–61. doi: 10.1002/eap.1627

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. (2015) 4:133–45. doi: 10.2217/cer.14.79

PubMed Abstract | CrossRef Full Text | Google Scholar