Mitigating Latent Threats Identified through an Embedded In Situ Simulation Program and Their Comparison to Patient Safety Incidents: A Retrospective Review

Objective: To assess the impact of service improvements implemented because of latent threats (LTs) detected during in situ simulation.

Design: Retrospective review from April 2008 to April 2015.

Setting: Paediatric Intensive Care Unit in a specialist tertiary hospital.

Intervention: Service improvements from LTs detection during in situ simulation. Action plans from patient safety incidents (PSIs).

Main outcome measures: The quantity, category, and subsequent service improvements for LTs. The quantity, category, and subsequent action plans for PSIs. Similarities between PSIs and LTs before and after service improvements.

Results: 201 Simulated inter-professional team training courses with 1,144 inter-professional participants. 44 LTs were identified (1 LT per 4.6 courses). Incident severity varied: 18 (41%) with the potential to cause harm, 20 (46%) that would have caused minimal harm, and 6 (13%) that would have caused significant temporary harm. Category analysis revealed the majority of LTs were resources (36%) and education and training (27%). The remainder consisted of equipment (11%), organizational and strategic (7%), work and environment (7%), medication (7%), and systems and protocols (5%). 43 service improvements were developed: 24 (55%) resources/equipment; 9 (21%) educational; 6 (14%) organizational changes; 2 (5%) staff communications; and 2 (5%) guidelines. Four (9%) service improvements were adopted trust wide. 32 (73%) LTs did not recur after service improvements. 24 (1%) of 1,946 PSIs were similar to LTs: 7 resource incidents, 7 catastrophic blood loss, 4 hyperkalaemia arrests, 3 emergency buzzer failures, and 3 difficulties contacting staff. 34 LTs (77%) were never recorded as PSIs.

Conclusion: An in situ simulation program can identify important LTs which traditional reporting systems miss. Subsequent improvements in workplace systems and resources can improve efficiency and remove error traps.

Introduction

Simulation is an important method for improving patient safety and quality of care (1) at individual, team, and system level (2). Team-based simulation can improve patient safety (3–8) by targeting team deficiencies, the greatest driver of lapses in patient safety (9). However, teamwork training alone will not overcome flaws in the fabric of the workplace. High reliability organizations (e.g., civil aviation) use in situ simulation to expose system vulnerability and hidden system flaws. These hidden threats are termed latent threats (LTs) (10). Proactive LT detection by in situ simulation is reported in emergency medicine, anesthesia, and pediatrics (11–13). In situ simulation is ideal for identifying LTs in the real work environment, using actual hospital systems, particularly during infrequent high stakes circumstances (14). Vulnerabilities in delivery settings deemed “safe” have been unveiled (11) and flaws in facility operational effectiveness detected (13, 15). In situ simulation offers a systematic and realistic picture of work because system flaws are contextualized in real time and place (3).

Patient safety incidents (PSIs) are unintended events with potential for patient harm, and all members of staff are encouraged to report these to Datix^®. PSI reporting is a statutory function of National Health Service (NHS) Improvement to reduce patient risks through the National Patient Safety Alerting System (16). Critiques of PSI reporting highlight a misplaced scrutiny on counting reports, rather than organizational learning from events (17, 18) and the difficulties of feeding-back to frontline staff (19–21).

There is a paucity of research on the effect of LT detection on traditional safety models such as Datix^®.

We hypothesized that LTs identified during in situ simulation courses could occur in real life; therefore, mitigating LTs through service improvements might improve patient safety. We aimed to measure:

1. The quantity, category, and subsequent service improvements of LTs.

2. The quantity, category, and subsequent action plans for PSIs (reported on Datix^®) occurring during the same period and including 1 year before the program inception in 2008.

3. The similarities between PSI Datix^® and LTs before and after service improvements.

Materials and Methods

The Simulated interPRofessional Team Training (SPRinT) program at Royal Brompton Hospital is an embedded in situ inter-professional simulation team training program that runs courses in all areas of pediatric care (22). SPRinT faculty are frontline multidisciplinary staff including nurses, nurse educators, anesthetic and intensive care consultants, and medical and surgical trainees. There is a minimum of one nurse and one doctor faculty member at each SPRinT course. Participants are all staff involved with pediatric care including all grades of medical, surgical, and anesthetic staff (consultant and junior doctors), nursing staff (charge nurse and band five to seven nurses), and allied health professionals (health-care assistants, family liaison officers, and physiotherapists). The validated in situ 2-h SPRinT session (23) includes crisis resource management training, a simulated scenario followed by 45 min video-assisted debriefing [utilizing advocacy–inquiry methodology (24)]. Scenarios are derived from real events including serious untoward incidents. Identification, investigation, and mitigation of LTs uncovered during courses are an integral part of SPRinT’s commitment to improving patient safety.

LTs and Service Improvements

During the debriefing after a simulation scenario, all faculty and participants identify any LTs that have occurred and suggest and discuss proposals for service improvement. These events are recorded at the time by two faculty members on a specific LT pro forma. LT severity is graded according to national Datix^® coding system. Datix^® is the online reporting system used by NHS organizations to report PSIs. They color code incidents from potential to cause harm (green), to causing minimal harm (yellow), to causing significant temporary harm (amber), and to permanent harm or death (red). LTs were uploaded to a SPRinT risk database from 2008 until February 2012 after which a specific LT reporting section on Datix^® was developed, the reporting criteria remained the same.

Latent threats recorded from Paediatric Intensive Care Unit (PICU) SPRinT courses from April 2008 to April 15 were retrospectively reviewed. SPRinT clinical fellows PK and HM independently categorized LTs according to seven descriptive categories: education and training, resources, equipment, work and environment, medication, organization and strategic, and protocols and systems. All differences were resolved by agreement.

Service improvements for LTs discussed during the SPRinT debriefing session are implemented by the SPRinT and department lead. All service improvements were quantified and grouped independently (HM and PK) according to nine categories: no change; looking into; ongoing quality improvement project (QIP); staff communication; guidelines; education; equipment/resources; organizational change; and all events reviewed by root cause analysis, serious case review, or serious hazard of transfusion.

PSIs and Action Plans

All PSIs on PICU reported to Datix^® from April 2007 to April 2015 were reviewed and independently categorized by Philip Knight and Helen MacGloin according to the same seven descriptive categories used for LTs. All differences were resolved by agreement.

Hospital risk management, quality, and safety departments organize regular review of Datix^® and subsequent action plans are made. These action plans are instituted by the lead for incident reporting for that department. Action plans for PSIs were quantified and grouped independently (Helen MacGloin and Philip Knight) according to the same nine categories used for service improvements.

Statistical Comparison

The quantity, category, and grading of LTs and PSIs were compared to assess any impact of LT service improvements on PSIs over time. This review was registered as a quality improvement initiative exempt from ethics review.

Results

Category Analysis of PSIs and Their Action Plans

1,946 PSIs were identified with severity grading of 1,405 (72%) with the potential to cause harm (green), 480 (25%) causing minimal harm (yellow), 60 (3%) causing significant temporary harm (amber), and 1 (0.05%) causing permanent harm or death (red). 1,140 (59%) PSIs could not be categorized because the precipitating cause could not be accurately identified, e.g., unanticipated surgery, initiation of extracorporeal membrane oxygenation.

Of 806 (41%) PSIs that could be categorized, medication PSIs accounted for 512 (64%), most reported errors in drug recording or controlled drug spillages. Other medication PSIs included prescribing errors, delayed or incorrect administration, or issues with expired and out of stock medications. 133 (17%) resource PSIs included missing resuscitation ventilator equipment, or infusion sets. 103 (13%) equipment PSIs included equipment failure requiring servicing or replacement. 29 (4%) work and environment PSIs included issues with arrest buzzers, blocked sinks, and bed-spaces. 13 (2%) PSI systems and protocols reports included provision of specialist services out of hours. Nine (1%) education and training PSIs included lack of knowledge of guidelines. Organization and strategic PSIs accounted for seven (0.9%) including incidents due to emergency admission pathways.

984 (51%) of all PSIs had 1,008 action plans; 962 (49%) reported only the event (without action plan). 622 (62%) involved staff communication (including individual staff spoken to, reminders, emails, signs safety and medicine bulletins, and patient safety newsletters). 132 (13%) planned to look into the event further; 104 (10%) planned new resources or fixed equipment; 40 (4%) planned organizational changes; 32 (3%) planned educational workshops, induction, changes to training or focused teaching; 32 (3%) planned to address the event within an existing QIP 24 (2%) new/updated guidelines; 22 (2%) were root cause analyses, serious untoward incidents, and serious hazards of transfusion.

Category Analysis of LTs and Service Improvements

201 SPRinT courses involved 1,144 inter-professional staff (488 doctors, 543 nurses, and 113 other allied professionals). 44 LTs were identified (1 LT per 4.6 courses) (Figure 1).

FIGURE 1

Figure 1. Latent threats over time.

44 LTs were uncovered and category analysis showed 16 (36%) related to resources, 12 (27%) education and training, 5 (11%) equipment, 3 (7%) organizational and strategic; 3 (7%) work and environment, 3 (7%) medication; 2 (5%) systems and protocols (Figure 2). LTs severity ranking of risk was 18 (40.9%) LTs with the potential to cause harm, 20 (45.5%) LTs that would cause minimal harm, 6 (13.6%) LTs that would cause significant temporary harm. LTs reached a peak in 2012 and then reduced over time year on year (Figure 1).

FIGURE 2

Figure 2. Latent threat category of risk.

43 subsequent service improvements were developed (2 LTs addressed by one service improvement); 4 (9.3%) of which were implemented trust wide (Tables 1 and 2).

TABLE 1

Table 1. LTs with the potential to cause harm.

TABLE 2

Table 2. LTs that would cause minimal harm and significant temporary harm.

There were three scenarios that were particularly high yield, uncovering multiple high-risk LTs. The first was a catastrophic blood loss (CBL) dual location SPRinT course in PICU and the Blood Transfusion Laboratory (BTL) involving three teams (anesthesia, PICU, and cardiothoracic surgery). Real-time simultaneous video-recording enabled a unique perspective on LTs at the PICU and the BTL interface. A 15-point PICU checklist was developed for this dual site simulation and its adherence was assessed (adherence 30%) and a 14-point checklist in BTL (adherence 100%). LT’s was uncovered including a delay contacting BTL technician, difficulty implementing emergency CBL management and collection of the wrong blood unit (Table 2). Following these events, emergency blood tracking requirements were changed so that issued blood units could be handed directly from BTL staff to PICU staff, rather than collection of blood units from the blood refrigerator.

Retesting of the CBL protocol during a second SPRinT course highlighted unfamiliarity with the new CBL protocol and failure to order all blood products (FPP, platelets, and cryoprecipitate). Subsequent service improvements included changes to mandatory staff training requirements, repeated simulation and refinement of CBL protocol with departmental education and subsequently implementation trust wide.

The second scenario involved an emergency resternotomy post cardiac surgery and uncovered five LTs that would have caused harm. They included the inability to perform a cardiac echocardiogram on a deteriorating simulated patient with cardiac tamponade, because the echo machine was in use elsewhere. Difficulty distinguishing adult and pediatric chest re-opening sets, difficulty in locating items on chest opening trolley, difficulty in contacting cardiothoracic surgeons, and delay in administering treatment due to time searching for drug cupboard keys (Table 2). Service improvements included a successful capital bid for an extra echo machine, purchase of mobile phone for nurse in charge and uploading consultants’ phone numbers, organizing automated drug cabinet and open chest trolley.

The third scenario was of hyperkalaemia progressing to ventricular fibrillation and cardiac arrest. LTs uncovered: lack of insulin (an essential part of emergency treatment), lack of awareness of the hyperkalaemia protocol, and lack of knowledge on correct use of the defibrillator. To mitigate the latter, in situ simulated electrophysiology workshops and defibrillation simulation scenarios reviewing external pacing and using internal defibrillation paddles were developed and 70 staff members participated (May 2011–February 2012).

Impact of LT Service Improvements on Future PSIs

Latent threats matched 24 (1%) of 1,946 PSIs. The impact of service improvements for these LTs was investigated.

The LT of poor management of hyperkalaemia was preceded by four hyperkalaemia PSIs of different causes. After service improvement (Table 1), one PSI occurred which was a record of the hyperkalaemia event with appropriate treatment and no adverse event.

The emergency buzzer failure LT was preceded by three similar PSIs with planned follow-up by the operational manager. After service improvement of implementation of regular system testing (Table 1), there were no further PSIs.

The two LTs of difficulty calling essential staff to the simulated emergency were preceded by two similar PSIs. In 2009, simulation participants were unable to contact on-call PICU consultants via switchboard, following which a mobile phone programmed with all the PICU consultant contact details was obtained for the PICU charge nurse. However, difficulties contacting other essential staff resulted in the same LT recurring, therefore the mobile phone contact list was updated. No further LTs or PSIs from difficulties in calling essential staff recurred.

Three CBL PSIs preceded the first simulated CBL scenario (Figure 3) although before this, CBL was not audited; therefore, these data are not robust. One of these CBL PSIs highlighted poor team communication, recommending SPRinT simulations to improve team communication and emergency chest opening training, which was subsequently carried out.

FIGURE 3

Figure 3. Paediatric Intensive Care Unit CBL Catastrophic Blood Loss Events (April 2007–2015) and impact of Simulation Led Service Improvements.

Two CBL PSIs occurred between the first and second CBL simulations in 2012. The first CBL PSI reported incorrect patient identification resulting in transfusion with blood intended for another child. The second PSI reported incorrect activation of CBL protocol. LT system improvements were implemented subsequent to the second CBL simulation, and in 2013 four CBL events occurred without PSIs. CBL PSIs in 2014 and 2015 reported difficulties contacting the blood transfusion technician because a CBL arrest call had not been properly activated.

Resource LTs (ETCO₂ lines and ECG paper) repeated as real events despite service improvements. Despite improved stock ordering after detecting lack of ETCO₂ lines, this was followed by three similar PSIs. The LT from lack of ECG paper was preceded by one similar PSI and despite service improvement, recurred as two PSIs.

34 LTs (77%) were never recorded as PSIs therefore the impact of the relevant service improvement could not be assessed. This encompassed all systems and protocols LTs (2/2); all equipment LTs (5/5); 10/12 (83%) education LTs; 13/16 (81%) resource LTs; 2/3 (66.7%) work and environment LTs; 1/3 (33%) medication, and 1/3 (33%) organizational LTs.

Discussion

In situ SPRinT courses identified LTs and implemented simple system adaptations rapidly with minimal cost or manpower. This echoes other in situ simulation studies reporting immediate risk mitigation and simple changes with few resources (12). In situ simulation coupled to system improvements is embraced by staff who witnessed the potential risk of LT to a real patient during simulation. Program engagement likely resulted from empowering staff to test hospital protocols, equipment, and environment for real system flaws. Such adaptive safety initiatives align with Berwick’s plea for frontline staff to be enabled to identify problems, test changes, and lead service improvements (25).

The majority of LTs were due to lack of resources and education and training deficits. Of the 12 LTs which recurred despite service improvements, half were due to lack of resources. Other LT studies similarly unveiled poorly understood policies, staff training deficiencies, and issues with critical materials and supplies (26, 27). In our review, seemingly improbable events during simulation (lack of insulin to treat hyperkalaemia and incorrect patient identification for blood transfusion) resurfaced in real life as PSIs. In addition, LTs uncovered significant training needs (e.g., defibrillation skills) unreported as PSI because education deficits are not well highlighted by incident reports.

Only 1% of PSIs were similar to LTs which might reflect the limitations of incident reporting and make PSIs poor measures of safety and quality improvement (28, 29). We also felt that because few of the LTs had corresponding PSIs this actually showed the unique value of simulation over patient safety reporting and are treating this as a positive finding. In addition, incident reporting and in situ simulation provide different vistas on safety. Other studies have highlighted the need for different approaches to build a comprehensive picture of safety because of the complementary yet different information provided by each approach (30).

Some repeat simulation scenarios enabled development and testing of protocols and a shared perspective ordinarily unavailable. For example, the dual location CBL simulations enabled a unique observation of simulated error at the interface of care between PICU and blood transfusion. This perspective resulted in an interdepartmental collaborative approach to patient safety, facilitated learning, and supported the rationale for system improvement. Although statistically impossible to prove CBL events are safer (due to rarity), CBL PSI grading lessened after system improvements. Likewise trust wide introduction of an intubation box reduced the time to intubate in all areas. Subsequently, excessive time to prepare for intubation has not recurred as an LT in simulated events. However, as never reported as a PSI, system improvement impact cannot be assessed, although it is obvious that reduction in time to intubate improves patient care in a high-risk environment.

Comparing medication LTs and PSIs is interesting. Although most PSIs reported medication errors, emergency drug dose calculations never featured as a PSI possibly because of one of the LT service improvements (emergency drug charts).

The traditional definition of safety (safety 1) is the absence of harm (31). However, safety is not just non-events but a proactive sensitivity to the possibility of failure (32). This deeper understanding of work-as-done enables anticipation of events and team and system flexibility and resilience (33). In situ simulation can encompass both safety perspectives. To the best of our knowledge, this is the first study to review LT detection over time, providing a rich reflection of work-as-done. Measuring LTs from simulation could be developed similarly by other trusts as a proxy safety metric, in conjunction with traditional retrospective PSI reporting, as complementary approaches to safety.

There were several limitations to this study. PSIs on Datix^® were reviewed to assess the impact of service improvements on safety. However, voluntary reporting of adverse incidents captures only a minority of incidents and is subject to significant bias (30). The possibility of bias and missing CBL events was recognized prompting review of the blood transfusion audit. PSI limitations could have been overcome by triangulating them with other records of patient safety events such as patient complaint records, risk management databases, and safety walk-rounds. Retrospective categorization was another source of potential bias. There is no standard nomenclature for error analysis on PICUs although frameworks for incident analysis could have been adapted (34). Other simulation studies have used failure modes effects analysis (FMEA) coupled to simulation training to prioritize predicted risk to enable solution development in stratified order (35). In contrast, we addressed LTs as detected, according to the perceived patient risk because some FMEA critiques highlight the lack of evidence base and the resource heavy methodology involved (36).

Iterative testing of system improvement efficacy was not performed which might have prevented later emergence of LTs as PSIs 2–3 years later. Follow-up interviews checking solution success might have enabled system improvement efficacy to be assessed (26). Another limitation was that the review was limited to PSIs and LTs from PICU, excluding LTs detected elsewhere in the directorate (e.g., PACU and the cardiac ward). Finally, participant psychological safety could have been threatened by reporting education and training LTs. Reassuringly, anonymous participant feedback never reported a negative impact on learning.

Conclusion

To the best of our knowledge, this is the first study comparing LTs identified during in situ simulation to existing safety reporting systems and evaluate subsequent service improvements. The unique strengths of in situ simulation were highlighted. First, the identification of potential threats to patient safety particularly for training and knowledge gaps undetected elsewhere. Second, in situ simulation offered a real-time unbiased multi-professional approach to patient safety.

Modern health-care pressures require better models of safety (34) and perspectives reflecting health care’s socio-technical system complexity. This cannot be achieved by one safety model but the best way to measure their individual or combined impact is unclear.

More work is required to integrate the strengths of in situ simulation into traditional models of patient care and safety and robustly measure its efficacy. This would enable in situ simulation to be harnessed by existing health-care systems and accepted as a valuable safety improvement modality.

Author Contributions

Participation in the conception and design of the work: MB, PK, and HM. Execution of the work: MB, LL, ML, CK, PK, AD, HM, EH, DM, and PM. Analysis of the data and contribution of methodological expertise MB, HM, and PK.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer EA and handling editor declared their shared affiliation.

References

1. Chief Medical Officer. Safer Medical Practice: Machines, Manikins and Polo Mints. 150 Years of the Annual Report of the Chief Medical Officer: On the State of Public Health 2008. London: Department of Health of the United Kingdom, HMSO (2009).

Google Scholar

2. Shear T, Greenberg S, Tokarczyk A. Does training with human patient simulation translate to improved patient safety outcome? Curr Opin Anesthesiol (2013) 26:159–63. doi:10.1097/ACO.0b013e32835dc0af

CrossRef Full Text | Google Scholar

3. Wheeler DS, Geis G, Mack EH, LeMaster T, Patterson MD. High-reliability emergency response teams in the hospital: improving quality and safety using in situ simulation training. BMJ Qual Saf (2013) 22(6):507–14. doi:10.1136/bmjqs-2012-000931

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Manser T. Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiol Scand (2009) 53(2):143–51. doi:10.1111/j.1399-6576.2008.01717.x

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Reader T, Flin R, Lauche K, Cuthbertson BH. Non-technical skills in the intensive care unit. Br J Anaesth (2006) 96(5):551–9. doi:10.1093/bja/ael067

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Donchin Y, Gopher D, Olin M, Badihi Y, Biesky M, Sprung CL, et al. A look into the nature and causes of human errors in the intensive care unit. Crit Care Med (1995) 23:294–300. doi:10.1097/00003246-199502000-00015

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Lightall G, Barr J, Howard S. Use of a fully simulated intensive care unit environment for critical event management training for internal medicine residents. Crit Care Med (2003) 31:2437–43. doi:10.1097/01.CCM.0000089645.94121.42

CrossRef Full Text | Google Scholar

8. Neily J, Mills PD, Young-Xu Y, Carney BT, West P, Berger DH, et al. Association between implementation of a medical team training program and surgical mortality. JAMA (2010) 304:1693–700. doi:10.1001/jama.2010.1506

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Pronovost P, Freischlag J. Improving teamwork to reduce surgical mortality. JAMA (2010) 304(15):1721–2. doi:10.1001/jama.2010.1542

CrossRef Full Text | Google Scholar

10. Reason J. The contribution of latent human failures to the breakdown of complex systems. Philos Trans R Soc Lond B Biol Sci (1990) 327(1241):475–84. doi:10.1098/rstb.1990.0090

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Blike GT, Christoffersen K, Cravero JP, Andeweg SK, Jensen J. A method for measuring system safety and latent errors associated with pediatric procedural sedation. Anesth Analg (2005) 101:48–58. doi:10.1213/01.ANE.0000152614.57997.6C

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Davis S, Riley W, Gurses AP, Miller K, Hansen H. Failure modes and effects analysis based on in situ simulations: a methodology to improve understanding of risks and failures. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 3: Performance and Tools). Rockville, MD: Agency for Healthcare Research and Quality US (2008).

Google Scholar

13. Gardner AK, Ahmed RA, George RL, Frey JA. In situ simulation to assess workplace attitudes and effectiveness in a new facility. Simul Healthc (2013) 8(6):351–8. doi:10.1097/SIH.0b013e31829f7347

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Riley W, Davis S, Miller KM, Hansen H, Sweet RM. Detecting breaches in defensive barriers using in situ simulation for obstetric emergencies. Qual Saf Health Care (2010) 19(Suppl 3):i53–6. doi:10.1136/qshc.2010.040311

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Kobayashi L, Shapiro MJ, Sucov A, Woolard R, Boss RM III, Dunbar J, et al. Portable advanced medical simulation for new emergency department testing and orientation. Acad Emerg Med (2006) 13:691–5. doi:10.1197/j.aem.2006.01.023

PubMed Abstract | CrossRef Full Text | Google Scholar

16. NHS Improvement Patient Safety Alert. Patient Safety Incident Reporting and Responding to Patient Safety Alerts. (2016). Available from: https://improvement.nhs.uk/uploads/documents/Patient_Safety_Alert_Stage_2_-_Patient_Safety_transfer_to_NHS_Improvement.pdf

Google Scholar

17. Davies HTO, Nutley SM. Developing learning organisations in the new NHS. BMJ (2000) 320:998–1001. doi:10.1136/bmj.320.7240.998

CrossRef Full Text | Google Scholar

18. Anderson J, Kodate N, Waters R, Dodds A. Can incident reporting improve patient safety? Healthcare practitioners views of the effectiveness of incident reporting. Int J Qual Health Care (2013) 25(2):141–50. doi:10.1093/intqhc/mzs081

CrossRef Full Text | Google Scholar

19. Gandhi T, Grayson-Baker E, Huber CN, Whittemore AD, Gustafson M. Closing the loop: follow-up and feedback in a patient safety programme. Jt Comm J Qual Patient Saf (2013) 31(11):614–21. doi:10.1016/S1553-7250(05)31079-8

CrossRef Full Text | Google Scholar

20. Benn J, Koutantji M, Wallace L, Spurgeon P, Rejman M, Healey A, et al. Feedback from incident reporting: information and action to improve patient safety. Qual Saf Health Care (2009) 18:11–21. doi:10.1136/qshc.2007.024166

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Frey B, Schwappach D. Critical incident monitoring in paediatric and adult critical care: from reporting to improved outcomes? Curr Opin Crit Care (2010) 16(6):649–53. doi:10.1097/MCC.0b013e32834044d8

CrossRef Full Text | Google Scholar

22. Stocker M, Allen M, Pool N, De Costa K, Combes J, West N, et al. Impact of an embedded simulation team training programme in a paediatric intensive care unit: a prospective, single-centre, longitudinal study. Intensive Care Med (2012) 38(1):99–104. doi:10.1007/s00134-011-2371-5

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Stocker M, Menadue L, Kakat S, De Costa K, Combes J, Banya W, et al. Reliability of team-based self-monitoring in critical events: a pilot study. BMC Emerg Med (2013) 13:22. doi:10.1186/1471-227X-13-22

CrossRef Full Text | Google Scholar

24. Rudolph JW, Simon R, Rivard P, Dufresne RL, Raemer DB. Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiol Clin (2007) 25:361–7. doi:10.1016/j.anclin.2007.03.007

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Berwick D. A Promise to Learn – A Commitment to Act: Improving the Safety of Patients in England. London: Department of Health (2013).

Google Scholar

26. Hamman WR, Beaudin-Seiler BM, Beaubien JM, Gullickson AM, Gross AC, Orizondo-Korotko K, et al. Using in situ simulation to identify and resolve threats to patient safety: case study involving a labor and delivery ward. J Patient Saf (2009) 5(3):184–7. doi:10.1097/PTS.0b013e3181b35e6c

CrossRef Full Text | Google Scholar

27. Patterson MD, Geis GL, LeMaster T, Wears RL. In situ simulation: detection of safety threats and teamwork training in a high risk emergency department. BMJ Qual Saf (2013) 22:468–77. doi:10.1136/bmjqs-2012-000942

CrossRef Full Text | Google Scholar

28. Levtizion-Korach O, Frankel A, Alcalai H, Keohane C, Orav J, Graydon-Baker E, et al. Integrating incident data from five reporting systems to assess patient safety: making sense of the elephant. Jt Comm J Qual Patient Saf (2010) 36(9):402–10. doi:10.1016/S1553-7250(10)36059-4

CrossRef Full Text | Google Scholar

29. Shojania K. The elephant of patient safety: what you see depends on how you look. Jt Comm J Qual Patient Saf (2010) 36:402–10. doi:10.1016/S1553-7250(10)36058-2

CrossRef Full Text | Google Scholar

30. Cullen DJ, Bates DW, Small SD, Cooper JB, Nemeskal AR, Leape LL. The incident reporting system does not detect adverse drug events: a problem for quality improvement. Jt Comm J Qual Improv (1995) 21(10):541–8.

PubMed Abstract | Google Scholar

31. Reiman T, Pietikainen E. Leading indicators of system safety – monitoring and driving the organisational safety potential. Saf Sci (2012) 50:1993–2000. doi:10.1016/j.ssci.2011.07.015

CrossRef Full Text | Google Scholar

32. Hollnagel E, Wears RL, Braithwaite J. From Safety-1 to Safety-2: A White Paper. Denmark, USA, Australia: The Resilient Healthcare Net, University of Southern Denmark, University of Florida, Macquarie University (2015). Available from: https://www.england.nhs.uk/signuptosafety/wp-content/uploads/sites/16/2015/10/safety-1-safety-2-whte-papr.pdf

Google Scholar

33. Pavard B, Dugdale J, Bellamine-Ben Saoud N, Darcy S, Salembier P. Underlying concepts in robustness and resilience and their use in designing socio-technical systems. Resilience Engineering Perspectives Vol 1: Remaining Sensitive to the Possibility of Failure. Ashgate Publishing Ltd. (2008). p. 1–18.

Google Scholar

34. Vincent C, Amalberti R. Safer Healthcare Strategies for the Real World. Cham, Heidelberg, New York, Dordrecht, London: Springer (2016).

Google Scholar

35. Lago P, Bizzarri G, Scalzotto F, Parpaiola A, Amigoni A, Putoto G, et al. Use of FEMA analysis to reduce risk of errors in prescribing and administering drugs in paediatrics wards: a quality improvement report. BMJ Open (2012) 2:e001249. doi:10.1136/bmjopen-2012-001249

CrossRef Full Text | Google Scholar

36. Dean Franklin B, Atef Shebl N, Barber N. Failure mode and effects analysis: too little for too much? BMJ Qual Saf (2012) 21:607–11. doi:10.1136/bmjqs-2011-000723

PubMed Abstract | CrossRef Full Text | Google Scholar