Longitudinal analysis of culture of patient safety survey results in surgical departments

Background: There is a need for improved methodologies on how to longitudinally analyze, interpret and learn from the Surveys on Patient Safety Culture™ (SOPS), developed by the Agency for Healthcare Research and Quality (AHRQ). Typically, SOPS quantify results by the percentage of positive responses, but this approach may miss insights from neutral or negative feedback.

Study design: The SOPS were distributed every two years from 2011 to 2022 to all hospital staff at one academic institution from perioperative services. Differences between rates of “positive” and “negative” scores (“Delta”), and “neutral” responses over time were calculated. The coefficient of determination (R²) was used to assess the correlation strength of the positive scores as the primary outcomes provided by the SOPS and Delta values over time. Finally, we evaluated patterns (crossing and converging [indicating “worrisome” patterns] vs. diverging [suggesting “desirable” pattern] vs. stable [suggesting “neutral” pattern]) of the longitudinal scores.

Results: A total of 1,035 responses were analyzed [51 and 40 survey items for SOPS v1 and v2 (2022 only), respectively]. Comparing the R² values of the positive only scores to the Delta scores demonstrated a change in effect size for “Nonpunitive Response to Error” (R² = 0.290 vs. 0.420). Of the 13 specific categories measured through SOPS, plotting negative vs. positive values elucidated 2 crossing, 2 converging and 2 diverging patterns indicating both a decrease in positive responses and an increase in negative responses rather than neutral.

Conclusion: Longitudinal analysis of the SOPS using the directional measures, Delta and pattern trends can provide organizations with additional key insights regarding culture of patient safety.

Introduction

Safety culture refers to the shared values, beliefs, and norms within an organization that influence behaviors and promote safety. In the context of healthcare, it encompasses the collective understanding of what is important regarding patient safety, as well as how systems operate within a healthcare organization (1–3). Patient safety culture is recognized as a crucial element in establishing and implementing patient safety programs. A few reasons that this is important is because patient safety culture assessment and improvement promote better patient outcomes, improved staff well-being, reductions in medical errors, and enhanced organizational effectiveness which can be appreciated on a global scale (4). Efforts to improve patient safety often prioritize the development of a strong safety culture within healthcare organizations. Some variables that define patient safety culture include overall patient safety perception, communication, leadership support, teamwork, staffing, and event reporting and analysis (5, 6). The fluid concept of safety culture can be difficult to measure as safety culture is defined based on a variety of variables, however, it has been demonstrated to be predictive of future performance across several accounts. Surgical teams have shown categorical and quantitative improvements after implementing tools like operating room checklists to help prepare for surgery, remember important information, and ensure medication safety (7–9). Overall, prior research highlights the importance of fostering a culture of safety using mechanisms such as event reporting, analysis, and continuous quality improvement efforts and how these actions contribute to attitudes of patient safety within hospital departments.

Within healthcare, the Agency for Healthcare Research and Quality's (AHRQ) Surveys on Patient Safety Culture™ (SOPS) is a commonly used tool to assess safety culture using questions to measure the functional categories of communication, teamwork, leadership, and overall perceptions of safety. Traditionally, the method of quantifying the results from the AHRQ survey is to assess the percent of positive responses. This conventional method may be suboptimal because it overlooks valuable insights from employees who perceive the safety culture negatively, remain neutral, or whose responses fluctuate over time. Nevertheless, past studies have shown the reliability, predictive validity, and psychometric soundness of the SOPS using surveys from hundreds of hospitals and found that it was reliable, valid, and psychometrically sound to use as an assessment tool for patient safety culture analysis (10–12).

We herein present an alternative approach to quantify the AHRQ survey that considers both the percent positive, and the percent negative, responses. Data gathered through the completion of the SOPS includes evaluations of overall patient safety perceptions and views on communication, leadership, and teamwork from the perspective of the healthcare personnel completing the survey. The intention of gathering this data is to demonstrate targetable areas for improvement within specific populations. This manuscript aims to explain an innovative method of analysis and the conclusions that can be drawn from the AHRQ SOPS data gathered over several years at a large academic institution within surgical departments. Ultimately, these conclusions and analyses could be used to understand the common problems within a large healthcare setting and assist in designing subsequent interventions to improve culture and patient safety outcomes.

Methods

Our institution distributed the AHRQ SOPS to all departments every two years from 2011 to 2019 and in 2022. Survey participation is voluntary, and responses are collected anonymously. Data analyses were conducted between December 2022 and April 2023. These six surveys over 11 years were filtered to include only the results collected from the 20 surgical departments/units/service lines representing surgical services. The surgical departments/units/service lines, provider roles, and work areas were used to select the data that can be seen in Table 1. The AHRQ made changes to the survey questions in the 2022 iteration of the SOPS (transition from SOPS v1 to v2), including removing individual questions, rewording questions, and relabeling result categories (13). Specifically, the final SOPS 2.0 has 40 survey items compared with 51 survey items in SOPS 1.0. The names of some composite measures were changed to align with changes to the content assessed in those measures. To compare the results of the 2022 survey to the surveys conducted between 2011–2019, individual questions for each survey in each category were matched to ensure analogous questions were compared across all surveys. The matched specific categories across surveys can be seen in Table 2. For each specific category included in the SOPS, the individual questions used to generate each category score each year were paired with the matching questions across all surveys. The percentage of positive responses (“Strongly Agree” or “Agree”) and negative responses (“Strongly Disagree” or “Disagree”) were calculated for each question and compared between years. The difference between the percentage positive and negative responses (Delta) was also calculated to compare shifts in the cultural perceptions of the survey population.

Table 1

Table 1. Data stratification of operating room departments from the AHRQ HSCS.

Table 2

Table 2. AHRQ SOPS questions categories and corresponding letter (A-N) legend.

The SOPS results were calculated by averaging the percentage of positive responses for each question within a category to generate a category score. To visually compare changes in reported SOPS results over time, these results were plotted by year to determine longitudinal changes. Additionally, the delta scores for each category were analyzed to determine the pattern (crossing and converging [indicating “worrisome” patterns] vs. diverging [suggesting “desirable” pattern] vs. stable [suggesting “neutral” pattern]) in positive vs. negative perceptions of survey respondents over time. Specifically, a crossing pattern implies that there is a change in direction of responses over time (if positive responses decrease while negative responses increase); a converging pattern suggests that positive and negative responses are moving closer together; a diverging response indicates that positive and negative responses are moving further apart; a stable pattern indicates a consistent perception. We sought to determine which domains within the SOPS showed the greatest change over time (in either the positive or negative direction) to inform design of interventions that will have the greatest impact on patient safety moving forward.

To compare the rate of change over all years analyzed in scores for each category, linear regression and the coefficient of determination (R²) were determined for each category. Utilizing the absolute R-squared values, the strength of a relationship for each value was determined using the following generalized strength categories: R-squared value <0.3 is generally considered a None or Very Weak effect size, an R-squared value 0.3 ≤ r < 0.5 is generally considered a Weak or Low effect size, a value 0.5 ≤ r < 0.7 is generally considered a Moderate effect size, and a value r ≥ 0.7 this value is generally considered Strong effect size (14, 15).

Results

Positive, negative and delta values over time are shown in Table 3. Linear regression coefficients and R² values for the positive-only and calculated Delta scores and corresponding patterns of the positive vs. negative responses over time are displayed in Table 4. For both analyses, the categories with the greatest declines in reported safety culture scores were “Patient Safety Rating/Unit Grade”, “Management Support for Safety”, “Learning from Errors”, and “Staffing”. The most improvement over the same period was seen in the category “Most Employees Report Events” in both analyses. Overall, R² values were only different in one category (“Nonpunitive Response to Error”), with positive vs. Delta results of 0.29 vs. 0.41, respectively.

Table 3

Table 3. Positive, negative, and delta results over time.

Table 4

Table 4. Coefficients of determinations (R²) for positive scores vs. delta values and emerging patterns.

Plots of the positive (% positive) and Delta (% positive—negative) scores are displayed in Figures 1, 2, respectively. In Figure 1, Panel A, the plot of positive scores demonstrates a gradual decline and weak effect size in both the “Overall Safety Score” and “Patient Safety Rating/Unit Grade”. The plot of Delta values in Figure 2, Panel A shows similar declines. Panel B in both Figures 1, 2 indicates that under the “Communication” category, the score for ‘Most Employees Report Events” was the only score to gradually increase over time with a strong effect size, while the “Nonpunitive Response to Error” demonstrated an increase in Delta values with a weak effect size, which was the only detected differences between the two plots and R² values. Panel C in Figure 2 demonstrates the “Handoff” scores on the Delta plot dipped below a score of 0 in 2015 and 2019 indicating more negative than positive responses. Lastly, in both figures, Panel D highlights that under “Leadership”, the scores for ‘Management Support for Safety” and “Staffing” demonstrated the most decline of all scores, with moderate and strong effect sizes, respectively. The “Staffing” scores on the Delta plot dipped below a score of 0 in 2019 and 2022 indicating more negative than positive responses.

Figure 1

Figure 1. Default AHRQ SOPS results presentation – positive scores only. (A) displays raw overall safety scores with patient safety rating/unit grade over time. (B) shows the reported frequency and types of communication observed over time. (C) displays reported teamwork scores across/within units vs during handoffs over time. (D) shows reported leadership scores appreciated over time.

Figure 2

Figure 2. Plotting of AHRQ delta values (% positive responses – % negative responses). (A–D) display the change in overall safety, communication, teamwork, and leadership scores, respectively, when accounting for the percentage of negative responses.

Discussion

Overall, the longitudinal analysis of the SOPS over an 11-year period within surgical departments provides valuable insights into the changing dynamics of safety culture. This study employed an innovative approach by considering both the percentage of positive responses and the calculated Delta values, shedding light on nuanced shifts in perception over time. The results from both analyses are similar but not the exact same; this suggests that the most concerning declines over time are in the perception of the patient safety rating at the unit-specific level, top management support for safety, and staffing levels. Similar findings have been reported by recent studies that found healthcare workers displayed a decrease in the perception of patient safety across most dimensions. Some of the contributory factors were lack of staffing, hospital management support, and organizational learning (4, 16–19). Another article found that staffing challenges characterized by extended working hours, high turnover rates, and dissatisfaction with workload and salaries also led to a decrease in perceived patient safety. These findings have been exacerbated by the COVID-19 pandemic due to increased employee workload and stress, shifts in priorities, fear of repercussions, communication challenges, and staffing changes (20–22).

The results are more easily notable in the analysis of the Delta scores (Figure 2), where all three mentioned categories neared or dipped below a score of zero. By calculating the Delta scores for each category, a more thorough and visual understanding of the survey results can be appreciated. For example, in Figure 2, the patient safety rating at the unit level in the year 2022 can be seen to decline to near zero (vs. a score of 32.2% positive responses in Figure 1), which indicates that the proportion of positive and negative responses is almost equal, representing a divergent pattern. This suggests that if left uncorrected, it may lead to a crossing pattern where most respondents now report negative perceptions. Similarly, using the same analogy, the crossing patterns can be noted for the top management support for safety and staffing categories. The comparison for Delta scores to positive-only scores highlighted a change in effect size for “Nonpunitive Response to Error”. This shift, from a very weak effect to a moderate effect, emphasizes the importance of considering both positive and negative responses in comprehensively understanding changes in safety culture.

The examination of overall safety culture positive scores revealed distinctive trends. The “Overall Safety Score” and “Patient Safety Rating/Unit Grade’ exhibited a weak trend with a negative effect size, suggesting a gradual decline in the perception of safety over the studied period. Similarly, “Management Support for Safety” demonstrated a negative trend with a moderate effect size, indicating a noteworthy decrease in positive sentiments regarding management support. The Leadership subcategory scores for “Staffing” and “Learning from Errors” exhibited a negative trend with a strong effect size, reflecting substantial declines in perceptions over time. Conversely, the “Communication” category specifically “Most Employees Report Events”, displayed a positive trend with a strong effect size, indicating a notable improvement in communication-related aspects of safety culture.

Interestingly, taking all the results in conjunction, the scores under the broader and specific categories remain relatively stable or convergent for 2011–2022, suggesting that despite the perceived decline in patient safety rating at the local level, top leadership support and staffing, the individual healthcare teams are relatively unchanged in the way they interact and perceive their roles, communication, and teamwork. This offers a potential narrative highlighting the grit and resilience of the surgical department workforce, especially given the challenges of the COVID-19 pandemic on the healthcare system (e.g., burnout, moral distress, trauma). Declines in patient safety ratings at the local level, top leadership support, and staffing can be partially associated with the COVID-19 pandemic (e.g., non-clinical leaders working from home, physicians and nurses leaving the profession, constant changes to clinical and operational protocols, less opportunity to learn from errors due to high workloads). Similar conclusions were noted by other studies conducted during the COVID-19 pandemic (23–25). One noticeable result from our study was that executive hospital leadership, while having high expectations from the healthcare workforce, was perceived to suboptimally provide the necessary support and empowerment that is crucial for maintaining patient safety and overall well-being of healthcare workers.

It is essential to address some inherent limitations that can influence the generalizability and applications of the data trends. First, the sample size, while met the minimum requirements for analysis per SOPS recommendations (13) comprised of data collected from surgical departments at a single academic institution is relatively limited and varier by year. We also recognize that the matching of questions between SOPS 1.0 and 2.0 is not trivial and could introduce some level of uncertainty in the combined analysis. While the dataset provides interesting insights into patient safety culture within surgical departments, it may not represent the diversity and complexity in healthcare nationally or globally. The dynamics and behaviors of patient safety culture can vary between institutions, regions, and specialties in healthcare. As a result, caution should be exercised when generalizing these results to other healthcare organizations. Second, even with a larger sample size, the study's exclusive focus on a single institution raises questions about using these results to create interventions that nurture a more cohesive patient safety culture in different healthcare settings. Other organizations vary widely in size, governance, and operational practices, and these differences can profoundly impact their patient safety culture. Lastly, it is important to mention that none of the improvement efforts developed and implemented through the course of this longitudinal project was considered or factored into our analysis. This serves as another limitation as it serves as a lead point for bias that can skew results.

The findings in this study lay the foundations for several other avenues of future research aimed at enhancing our understanding of patient safety culture within healthcare organizations and developing more effective interventions. Building on the methodology employed in this study, future research could delve deeper into the dynamics of patient safety culture by conducting Delta analyses across different organizational roles within healthcare settings. For instance, separate analyses could be performed for top management, local management, physicians, nurses, and other healthcare professionals. This granular approach can reveal nuanced variations in how different groups perceive and experience patient safety culture. Understanding each role's specific concerns, strengths, and weaknesses can guide targeted interventions and improvements. This data could offer a more comprehensive view of cultural trends within institutions. The current study, while limited to one academic institution, emphasizes the need for broader investigations across a more extensive range of healthcare organizations. Future studies should consider the incorporation of larger and more diverse datasets from other healthcare settings (i.e., hospitals, clinics, etc.). By including data from different healthcare systems and regions, researchers can identify commonalities and distinctions in trends of patient safety culture. Moreover, large-scale studies have the potential to identify overarching patterns over time.

Conclusions

Overall, this longitudinal study of SOPS within surgical departments over an 11-year period offers valuable insights into the evolving dynamics of patient safety culture at a large academic institution. By utilizing an innovative approach that considers both positive and negative responses through Delta scores and analyzing longitudinal patterns, the study reveals important shifts in perceptions that may not be captured by traditional methods focused solely on the percentage of positive response rates. The findings indicate that while certain aspects of safety culture, such as communication and reporting, have shown improvement over time, notable declines were observed in areas critical to patient safety, including “Management Support for Safety”, “Staffing”, and the overall “Patient Safety Rating”. The Delta scores highlight these concerning trends more clearly, particularly where positive and negative responses have converged or crossed, signaling areas in need of further exploration.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving humans were approved by the University of North Carolina at Chapel Hill Institutional Review Board (IRB). The studies were conducted in accordance with the local legislation and institutional requirements. The Ethics Committee/institutional review board waived the requirement of written informed consent for participation from the participants or the participants' legal guardians/next of kin because the participants gave verbal consent prior to participation, as required for IRB approval.

Author contributions

LB: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing – original draft, Writing – review & editing. SL: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing – original draft, Writing – review & editing. CM: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing – original draft, Writing – review & editing. JR: Conceptualization, Data curation, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. CG: Writing – review & editing. LaM: Writing – review & editing. TI: Writing – review & editing. LuM: Conceptualization, Data curation, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. Funding provided by Innovate Carolina.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Pumar-Mendez MJ, Attree M, Wakefield A. Methodological aspects in the assessment of safety culture in the hospital setting: a review of the literature. Nurse Educ Today. (2014) 34(2):162–70. doi: 10.1016/j.nedt.2013.08.008

PubMed Abstract | Crossref Full Text | Google Scholar

2. Bellot J. Defining and assessing organizational culture. Nurs Forum. (2011) 46(1):29–37. doi: 10.1111/j.1744-6198.2010.00207.x

PubMed Abstract | Crossref Full Text | Google Scholar

3. Carney M. Influence of organizational culture on quality healthcare delivery. Int J Health Care Qual Assur. (2011) 24(7):523–39. doi: 10.1108/09526861111160562

PubMed Abstract | Crossref Full Text | Google Scholar

4. Lawati MH, Dennis S, Abdulhadi NN. Patient safety and safety culture in primary health care: a systematic review. BMC Fam Pract. (2018) 19:104. doi: 10.1186/s12875-018-0793-7

PubMed Abstract | Crossref Full Text | Google Scholar

5. DiazGranados D, Dow AW, Appelbaum N, Mazmanian PE, Retchin SM. Interprofessional practice in different patient care settings: a qualitative exploration. J Interprof Care. (2018) 32(2):151–9. doi: 10.1080/13561820.2017.1383886

PubMed Abstract | Crossref Full Text | Google Scholar

6. Gershon R, Stone P, Bakken S, Larson E. Measurement of organizational culture and climate in healthcare. J Nurs Admin. (2004) 34:33–40. doi: 10.1097/00005110-200401000-00008

PubMed Abstract | Crossref Full Text | Google Scholar

7. Harris K, Softeland E, Moi AL, Harthug S. Patients’ and healthcare workers’ recommendations for a surgical patient safety checklist—a qualitative study. BMC Health Serv Res. (2020) 20:43. doi: 10.1186/s12913-020-4888-1

PubMed Abstract | Crossref Full Text | Google Scholar

8. Haugen AS, Softeland E, Sevdalis N, Eide GE. Impact of the Norwegian national patient safety program on implementation of the WHO surgical safety checklist and on perioperative safety culture. BMJ Open Qual. (2020) 9:162–70. doi: 10.1136/bmjoq-2020-000966

Crossref Full Text | Google Scholar

9. Mazur L, Chera B, Mosaly P, Taylor K, Tracton G, Johnson K, et al. The association between event learning and continuous quality improvement programs and culture of patient safety. Pract Radiat Oncol. (2015) 5(5):286–94. doi: 10.1016/j.prro.2015.04.010

PubMed Abstract | Crossref Full Text | Google Scholar

10. SOPS Surveys. Content last reviewed June 2024. Rockville, MD: Agency for Healthcare Research and Quality (2018). Available online at: https://www.ahrq.gov/sops/surveys/index.html (accessed October 10, 2024).

Google Scholar

11. Etchegaray JM, Thomas EJ. Comparing two safety culture surveys: safety attitudes questionnaire and hospital survey on patient safety. BMJ Qual Saf. (2012) 21(6):490–8. doi: 10.1136/bmjqs-2011-000449

PubMed Abstract | Crossref Full Text | Google Scholar

12. Sorra JS, Dyer N. Multilevel psychometric properties of the AHRQ hospital survey on patient safety culture. BMC Health Serv Res. (2010) 10:199. doi: 10.1186/1472-6963-10-199

PubMed Abstract | Crossref Full Text | Google Scholar

13. Agency for Healthcare Research and Quality. Hospital Survey on Patient Safety Culture: User’s Guide. Rockville, MD: AHRQ (2019). Available online at: https://www.ahrq.gov/sites/default/files/wysiwyg/sops/surveys/hospital/hsops2-pt1-transition-updated.pdf (accessed July 19, 2024)

Google Scholar

14. Kelley K, Preacher KJ. On effect size. Psychol Methods. (2012) 17:137–52. doi: 10.1037/a0028086

PubMed Abstract | Crossref Full Text | Google Scholar

15. Moore DS, Notz WI, Flinger MA. The Basic Practice of Statistics. New York, NY: W. H. Freeman and Co (2013).

Google Scholar

16. Almutairi S, Aljohani A, Awad M, Saha C, Alhobaishi H, Almutairi A, et al. Perceptions of patient safety culture dimensions among hospital nurses: a systematic review. DSAHMJ. (2022) 4(3):103–25. doi: 10.1007/s44229-022-00012-z

Crossref Full Text | Google Scholar

17. Hellings J, Schrooten W, Klazinga NS, Vleugels A. Improving patient safety culture. Int J Health Care Qual Assur. (2010) 23(5):489–506. doi: 10.1108/09526861011050529

PubMed Abstract | Crossref Full Text | Google Scholar

18. Hyatt RD. Nurse perceptions: the relationship between patient safety culture, error reporting and patient safety in U.S. hospitals [dissertation]. Franklin University, Columbus, OH, United States (2020).

Google Scholar

19. Reis CT, Paiva SG, Sousa P. The patient safety culture: a systematic review by characteristics of hospital survey on patient safety culture dimensions. Int J Qual Health Care. (2018) 30:660–77. doi: 10.1093/intqhc/mzy080

PubMed Abstract | Crossref Full Text | Google Scholar

20. Brborovic O, Brborovic H, Hrain L. The COVID-19 pandemic crisis and patient safety culture: a mixed-method study. Int J Environ Res Public Health. (2022) 19:2237. doi: 10.3390/ijerph19042237

PubMed Abstract | Crossref Full Text | Google Scholar

21. Pimenta Lopes Ribeiro OM, de Lima Trindade L, Silva Fassarella C, de Abreu Pereira SC, Figueiredo Cabral Teles PJ, Gomes da Rocha C, et al. Impact of COVID-19 on professional nursing practice environments and patient safety culture. J Nurs Manag. (2022) 30(5):1105–14. doi: 10.1111/jonm.13617

PubMed Abstract | Crossref Full Text | Google Scholar

22. World Health Organization. Implications of the COVID-19 Pandemic for Patient Safety: A Rapid Review. Geneva: World Health Organization (2022). Available online at: https://iris.who.int/bitstream/handle/10665/361283/9789240055094-eng.pdf?sequence=1 (accessed October 10, 2024).

Google Scholar

23. Denning M, Goh ET, Scott A, Martin G, Markar S, Flott K, et al. What has been the impact of COVID-19 on safety culture? A case study from a large metropolitan healthcare trust. Int J Environ Public Health. (2020) 17(19):7034. doi: 10.3390/ijerph17197034

Crossref Full Text | Google Scholar

24. Idilbi N, Dokhi M, Malka-Zeevi H, Rashkovits S. The relationship between patient safety culture and the intentions of the nursing staff to report a near-miss event during the COVID-19 crisis. J Nurs Care Qual. (2023) 38:264–71. doi: 10.1097/NCQ.0000000000000695

PubMed Abstract | Crossref Full Text | Google Scholar

25. Lotfi M, Akhuleh OZ, Judi A, Khodayari M. Turnover intention among operating room nurses during the COVID-19 outbreak and its association with perceived safety climate. Perioper Care Oper Room Manag. (2022) 26:100233. doi: 10.1016/j.pcorm.2021.100233

PubMed Abstract | Crossref Full Text | Google Scholar