Assessing the Impact of Culture: A Systematic Analysis of Culture-Based Interventions and Evaluations

Over the last 20 years, transportation agencies have added culture-based approaches to the existing education, engineering, and enforcement (3E) strategies being used as a means of reducing traffic related injuries and fatalities. Despite this increased interest, there have been comparatively few evaluations of these interventions. At the same time, many other organizational types have adopted culture-based strategies either to improve safety or to enhance other elements of organizational performance. In aggregate, the evaluations of culture-focused interventions across a range of settings offer an untapped body of information about the models of culture being leveraged to affect change, the intervention strategies used to impact culture, the impacts of these strategies, and more. The research presented here used a systematic analysis of published, peer-reviewed evaluations of culture-based interventions designed to enhance traffic safety, safe workplaces, and effective practices more broadly, in order to more comprehensively understand the current state and impacts of culture-based theory and practices. This study seeks to build a clearer picture of the presence and robustness of current models of culture being utilized, as well as what evaluation strategies and designs are being used to assess culture-based initiatives, and further, what the robustness of the existing models and evaluation designs reveal about the efficacy of the original, culture-based interventions. Lastly, this analysis includes a preliminary examination of links between culture-based initiatives and systems theory/thinking, in an effort to identify conceptual ties between and/or empirical evidence supporting a relationship between the “Safe System” approach and traffic safety culture or other culture-based efforts. It is hoped that with a clearer understanding of the status of the field, it will be possible to make further progress in crafting, assessing, and refining theoretically robust and empirically grounded efforts to advance the efficacy of traffic safety culture and safe systems approaches.

Introduction

Over the last decades, a substantial number of traffic safety agencies have developed and implemented new initiatives that seek to change both agency and road user culture to reduce the number of injuries and fatalities on public roads and highways. As the number and use of culture-based safety initiatives has increased, systematic evaluations of those operations and their impacts have not kept up with the programs themselves. A growing number of researchers have noted that road safety campaigns - one type of strategy used to change traffic safety culture - are rarely subjected to a formal and complete evaluation (Hoekstra and Wegman 2011; Robertson and Pashley 2015). This lack of accessible evaluation data severely restricts the advancement and adoption of effective campaigns because there is 1) no guidance on how to improve campaigns, 2) no evidence to discontinue ineffective campaigns, and 3) no impetus to advance safety campaign techniques. Both peer-reviewed and professional literatures suggest that there is a consistent set of barriers to both conducting evaluations and using the results in instances when evaluations are conducted. Commonly cited barriers include factors such as a lack of time and resources, insufficient knowledge to conduct or use evaluations, and skeptical attitudes among program staff about the process and results of evaluations (Brescianai, 2011; Holosko, 1996). The U.S. General Accountability Office (GAO) reports that less than 40% of the federal agencies they examined in the United States had conducted formal evaluations of their programs. However, 80% of the agencies that did conduct evaluations reported multiple benefits from having done so (Government Accountability Office 2013). There is, however, a more extensive body of literature that evaluates other safety culture and organizational culture interventions outside of traffic safety. As is often the case, disciplinary specificity has prompted researchers and practitioners alike to retain a fairly narrow focus on what is known within their specific discipline and tends to give little attention to what may be gleaned from other fields of study. As a result, there has yet to be a broader examination of culture change initiatives across disciplines and settings. In an effort to close this gap, this study presents the results of a systematic analysis of evaluations conducted on traffic safety culture initiatives, as well as evaluations of safety culture and organizational culture change in other industries and settings.

Evolution of Culture and Systems Theories and Their Application

Although the concept of culture has become increasingly adopted across a range of academic disciplines, has been utilized in a variety of settings, and has intuitive appeal, it has also been critiqued for being insufficiently clear and precise (Cox and Cox 1996; Hale 2000). To provide as much clarity as possible, it is useful to first ground and locate the approach to culture being deployed here. Anthropologist Clifford Geertz was among the first scholars to develop and operationalize a definition and systematic approach to the study of culture. Geertz, in his seminal book The Interpretation of Cultures, describes culture as, “a system of inherited conceptions expressed in symbolic forms by means of which men communicate, perpetuate, and develop their knowledge about and attitudes toward life” (Geertz 1973). Despite the fact that such inherited conceptions reside in the minds of individuals who are a part of any culture, that culture, according to Geertz, is public in that its expression is manifest in patterns of social interaction. Not surprisingly, recognition of the presence and function of cultural attributes such as the communication, development, and perpetuation of knowledge and attitudes quickly moved from anthropology to other disciplines and was recognized in narrower, more specific settings including organizations. Organization theorists had long recognized the limitations of both direct supervision and the use of rules and procedures as the sole basis of managing employee performance (see, for example, the work of Luther Gulick and Urwick (1937) or Herbert (Simon, 1976)). The first clear indicators of those limits emerged with the Hawthorne Experiments during the 1930s (Roethlisberger and Dickson 1939), the behavioral revolution of the two decades following World War II yielded a wide range of models that sought to better understand and describe nonrational (i.e. exceeding both economic, self-interested and utility focused model of behavior, and Simon’s boundedly rational or satisficing model) forms of behavior. The behavioral revolution resulted in the development of a diverse range of motivational, structural and cognitive strategies to support behavior change. Later still, in the 1980s and ’90s, organization theorists, drawing from anthropology, began to explore not only the development and perpetuation of cultural values, beliefs, and their functions in organizations, but how those attributes could affect employee actions and ultimately the performance of organizations as a whole. The intentional development and management of organizational culture, it seemed, could be used as a means of establishing a set of shared perceptual attributes including values and beliefs that, especially when coincident with the organization’s policies, procedures, mission and objectives, could enhance individual and ultimately organizational effectiveness (e.g. Schein 2004). While these cultural approaches retained much of the behavioral revolution’s perspective on nonrational forms of behavior, they differed in that the unit of analysis associated with cultural theories and interventions tends to focus primarily on the structures of and within groups, whether a society, organization, or user group, rather than the perceptual, experiential or motivational drivers of the individuals who comprise the group.

Safety Culture

As research on organizational culture has evolved, researchers and practitioners alike have refined and developed greater specificity in the application of culture to particular settings and concerns that include the articulation of managerial priorities, the availability and distribution of resources, and the development of policies and procedures that support–or inhibit–consistency with articulated values (Nieva 2003). Among these more specific areas of focus, safety culture or organizational safety culture (OSC) has emerged as a concept relating more narrowly to the beliefs and values concerning health and safety within an organization and the degree to which those attributes are embodied in practices and expressed in performance (Clarke 1999). The rationale for considering culture as a lever for improving workplace safety can be recognized in the notion that a three-pronged approach to safety focusing on an improved physical environment, and/or education and enforcement (3E) was necessary, but not sufficient to achieve safety standards by itself. However, achieving those safety standards could be augmented by simultaneously building safety-oriented beliefs and values into the culture of the setting, and thereby inculcating those values and beliefs into the individual and collective behavior of members. OSC has been used as a contributing element of a wide range of organizational analyses (Cox and Flin 1998) and intervention initiatives designed to make the workplace less risky (Luria and Rafaeli 2008). Definitions of organizational as well as safety culture, however, have remained variable and often ambiguous. Among those that move toward operational levels of detail, Reason (2000), for example, argues that safety culture expresses the “ability of individuals or organizations to deal with risks and hazards so as to avoid damage or losses and yet still achieve their goals” (p. 5). More recently, OSC has been described as the “assembly of underlying assumptions, beliefs, values and attitudes shared by members of an organization, which interact with an organization’s structures and systems and the broader contextual setting to result in those external, readily-visible, practices that influence safety” (Edwards et al., 2013). As a result of the parallels between OSC and traffic safety culture, there has been some adaptation or adoption of cultural indicators from the former into the later, such as culturally-based beliefs, values and behaviors. Despite ongoing efforts to further the operationalization and specificity of culture models, others, including Cox and Cox (1996) and Hale (2000), have critiqued the abstract and conceptual character of safety culture and raised concerns about the clarity, precision, and utility of the concept. Despite, and to some degree in response to these concerns (Havold and Nesset 2009), researchers and practitioners have continued to extend and enhance the use of OSC.

Traffic Safety Culture

Given the evolution of theories of culture, organizational culture, and now organizational safety culture, it is little surprise that a cultural approach has also made its way into traffic safety. Moreover, given that limits of 3E or related approaches in traffic safety are so similar to OSC, the potential to enhance traffic safety culture by building systems that contribute to safety-oriented values and beliefs–i.e. culture - has gained significant traction. Among the earliest instances of this trend can be found in the 2007 AAA report, which provided an initial outline (drawing from OSC) of what traffic safety culture is, including models and related indicators of culture, and a call to action for researchers and practitioners in this nascent field (Hedlund 2007). That initial interest continued to grow in the subsequent years, as is evidenced by the breadth and diversity of efforts described during the 2011 Transportation Research Board sponsored conference in Washington DC (Turnbull 2011). Efforts to further develop and refine both the concepts and practices associated with traffic safety culture continued to appear throughout the last decade, including pieces by (citation removed for review) and Edwards et al. (2014). Some of the most recent work provides a more detailed model and etiology of traffic safety culture and user behavior (citation removed for review) and an extensive description of the attributes or indicators of culture, starting with three categories of beliefs, behavioral, normative and control beliefs respectively, and subsequently broken down further into their attitudinal expressions, recognizable norms and ideals, and embodied perceptions of control (citation removed for review).

Open and Socio-Technical Systems

Concurrent with, though largely independent of the developing culture-based thinking described in the previous section, is the emergence of the “safe systems” approach to traffic safety that, as described by the ITE, is aimed at designing and operating both vehicles and infrastructure in a way that anticipates human error and accommodates human injury tolerances with a goal of reducing fatal and serious injuries (Abel et al., 2020). This safe systems approach draws heavily on understandings of the behavior of socio-technical systems. This line of thinking developed at more-or-less the same time as much of the work on organizational culture, and there are two post-World War II lines of systems thinking to note. The first is open systems theory as described by (von Bertalanffy 1950) around natural systems, which was subsequently operationalized into organizations by organizational theorists like (Katz and Kahn 1978). The second shared line of influence begins with the work on statistical quality control as it developed into approaches like Total Quality Management (Walton 1986) and subsequently into systems-oriented learning organizations (Senge 1990). This line of also contributes to the evolution of socio-technical systems approaches that are concerned with the interaction between humans and various machines or other technologies. (Perrow, 1984) Normal Accidents and later, Rasmussen (1997) influential piece on risk management in complex environments are foundational to establishing the grounding and framing for the safe systems approach. While there are conceptual connections between cultural and systems theories and a substantial collection of literature that recognizes these connections, the intersection as gone largely without extensive development and there are relatively few works that endeavor to substantively and strategically synthesize the two sets of ideas (one notable exception is Harrison and Corley 2011).

Problem and Research Question

There is widespread and growing recognition within and beyond the field of traffic safety that the effectiveness of interventions focused on a 3E or related approach have limits and can benefit from the addition of other change strategies. Culture-based approaches have gained increasing attention and are being adopted in a variety of forms and across a wide variety of settings. Nevertheless, there is a great degree of variation in the understandings of culture used to inform these approaches and, correspondingly, disparity in how those interventions are assessed. These variations and disparities pose a challenge to practitioners wanting to adopt culture-based strategies, as they seek to identify practices that are most likely to accomplish their programs’ objectives.

In an effort to build more continuity and deeper, shared understandings of culture-based theories, their use, and implications, this study examines what a systematic analysis of the organizational and safety culture literature reveals about:

• The current models and corresponding uses of cultural theory being utilized across fields.

• The indicators and measures used to assess culture and culture change.

• The strategies and techniques to assess those interventions.

• The outcomes revealed by existing assessments.

Materials and Methods

The research design used for this study follows the approach described by Kapucu et al. (2017). We describe it here as a systematic analysis in that it utilizes a more rigorous examination of a body of literature than a traditional literature review (Ham-Baloyi et al., 2016), but it does not aggregate and analyze the data gathered by included studies as would be the case with a meta-analysis. Nevertheless, the approach to systematic analysis used here enables researchers to perform a more substantive and sophisticated assessment of patterns and relationships within a body of literature than a traditional literature review. In this study, we seek to develop a better understanding of what we can discern from the existing literature about culture change initiatives, the approaches and culture models used, the effectiveness of those efforts, and how effectiveness is evaluated in ways that improve the understanding of both researchers and practitioners.

Data Collection

To maximize the consistency and accessibility of the literature examined in this study, we chose to use peer reviewed journal articles, rather than professional publications or books. To identify articles for the project, a keyword search was performed using Montana State University Library’s “CatSearch” meta-search engine. This search engine consolidates access to all of the University’s databases including InfoTrac, Academic Search Complete, JSTOR, Lexis-Nexis, and others into one comprehensive search engine. Initially, keyword and phrase combinations were used as the basis of broad searches for “traffic safety,” “traffic safety culture,” “transportation safety,” and “transportation safety culture.” This search was augmented with a follow-on search of specific search engines that previously yielded the largest number of results including ProQuest Central, Elsevier Science Direct Journals Complete, and Emerald A-Z Current Journals to corroborate the results from CatSearch. Lastly, because the focus of this research is to better understand what is known about culture-based approaches to improve traffic safety, our search also included the Transportation Research Board’s Transportation Research International Documentation (TRID) and Research in Progress (RIP) databases.

Searches were structured to include only results published in English and included studies that were conducted both in and outside the United States. While the search strategy did not use any date parameters to restrict the search, only four articles published before 2000 were part of the initial search results. As expected, based on a previous, preliminary examination of the literature, this search did not yield studies focused on the implementation and evaluation of traffic safety culture change initiatives. As a result, the search and screen process was shifted to be more inclusive, utilizing new keyword and phrase searches to expand the search outside traffic and transportation safety domains to capture published evaluations of other safety culture and organizational culture change initiatives. Search terms including “safety culture” and “culture change” were used for both title and subject searches, in combination with additional key words including “intervention,” “evaluation,” and other variations. The rationale for this approach is premised on the notion that while the content of any given culture, including specific values and beliefs, will vary from one setting to the next, and the indicators or artifacts of that culture will vary correspondingly, the function of culture relative to its influence on the behavior of those in any specific setting will be structurally similar. As a result, this broader search allowed us to focus on culture change initiatives that have been both implemented and evaluated in other disciplines and organization types, which let us focus more widely on culture change initiatives, their implementation, and any results attributed to the interventions. Once an initial set of articles was generated by the search engine, each article was reviewed by two members of the research team to ensure that the final sample included only empirical evaluations and assessments of culture-based interventions, and excluded, for example, articles like those making a theoretical case for conducting evaluations of such interventions. The search and review of materials resulted in a final set of 59 articles that were then analyzed, of which, three were specifically focused on culture-based traffic safety programs. Figure 1 provides a summary flow chart of the search and screening steps used to yield the data set, and is based on the PRISMA flow diagram (Moher et al., 2009).

FIGURE 1

FIGURE 1. Search and screening process.

Analysis

Once the sample of articles was complete, PDF files for all of the articles identified were imported to the qualitative data management and analysis software NVivo for coding and analysis. The articles were coded using a combination of open and axial coding. Unlike a more typical grounded theory study (Strauss and Corbin 1990) where no pre-existing theory provides a framework for coding, this study began with a preliminary set of propositional codes that had been developed from existing literatures. A preliminary coding scheme that included features such as research design, analysis, and setting or industry, was augmented with additional, open codes that emerged during the coding process.¹ The coding scheme resulted in a two-level hierarchy of codes called parent and child nodes. For example, the parent node for research design had a number of child nodes including quasi-experimental, single-group mixed methods, qualitative, etc. This hierarchy of parent and child nodes enabled the researchers to more easily structure the analysis at later stages of the project.

In an effort to enhance the reliability of the coding process, several of the first articles were coded independently by two researchers and then reconciled. That reconciliation was used to ensure clarity and consistency in the coding process going forward. Subsequently, a second selection of articles was chosen at the later stages of the coding process, and the coding completed by one researcher for those articles was again reconciled with that of a second researcher to ensure that the use and understanding of the codes had not diverged throughout the course of the coding process. Finally, the list of references that resulted from the process coding were reviewed for consistency.

The analysis of the coded materials included three main elements. The first involved an examination of the patterns that emerged from the coding process both within and across parent and child nodes. The examination of patterns included consideration of various codes’ frequencies, both those that were more frequent and also those that were unique or unusual.

The second element of the analysis involved conducting a series of word frequency queries. These queries allow us to look at how frequently words appeared in the entire data set, but also within categorical subsets of the data. These queries are a means of identifying differences and similarities in focus and emphasis, for example, across industries. That is, by doing a word frequency query within each industry, we are able to get some indication of what is emphasized or prioritized within an industry and how that might vary between industries. Although, as we noted above, the specific cultural values, beliefs, indicators and behaviors can vary from one culture or setting to the next, the concept of culture suggests that its attributes and functions should be self-similar regardless of specific setting or the specific expression of a culture in the values, language or other features. Moreover, because the sample for this study was drawn from peer-reviewed, English-language journals, we are also assuming a degree of conceptual and terminological consistency within the sample. So, while the perceptions and languages among members of different cultures can and will differ, this analysis is not focused on, nor does it utilize data that is likely to substantively vary because of those contextual differences.

The final element of the analysis involved a series of matrix or cross-tab queries. These queries allow for the identification of patterns in the relationships or intersection of different parent and child nodes. For example, a query that compares the intersection of all the child nodes within the Culture Theory node (i.e. the categories of or cultural theories used by each study) with industry allows us to see if certain industries or sectors tend to use any one particular cultural/theoretical framework in their interventions or analyses by comparison to another industry.

Results

The results of our analysis are presented in three categories. The first focuses on the models and theories of culture identified in the literature we analyzed and related patterns that emerged. The second area of analysis focuses on prospective patterns in the literature related to the industry or organizational type (e.g. health care, education, transportation, etc.) studied. The final area of focus is on the evaluation designs and analytical approaches used by the literature we analyzed.

Theories and Models of Culture

The first area on which the analysis focused was the theories or models of culture adopted by the initiatives being evaluated in this body of literature and any patterns that emerged regarding the links between the models of culture and:

• Outcome indicators or measures of culture change or of program impact.

• The unit of analysis used by the study and the target of the intervention.

• Impacts of the intervention or change initiative being evaluated.

The working hypothesis with which we began the analysis was that the cultural theories or models used by change agents would fall into categories associated with the major organizational culture theories. Our initial assumption was that if we could identify those theoretical orientations, we could begin to look for patterns that link those orientations to indicators and measures of change, and even impacts of the interventions. Figure 2 below is the “node tree map” that shows the “child” or sub-codes that fell under the overarching “Culture and Other Theory” node. These categories resulted from the process of axial and open coding for culture theories. The size of each child node box within the node-tree map represents the relative frequency of each theory we identified in the data. As we anticipated, Edgar Schein’s organizational culture model was common. However, the two other child nodes for culture, “Culture-Other” and “No Clear or Explicit Model,” were created because no other specific model of culture revealed itself in the literature. The “Culture-Other” node was created for those articles that had a specific definition or description of culture, but that could not be traced to a recognized organizational culture source (e.g., Westrum) or other cultural theory source (e.g., Clifford Geertz). The “No Clear or Explicit Model” was created for those articles that note a cultural approach but that do not define, describe, or otherwise articulate a specific understanding of what culture is or how it precisely functions in a way that can be linked to an identifiable cultural theory.

FIGURE 2

FIGURE 2. Culture and other theory node tree map.

The final child node that should be noted in Figure 2 is the “Learning-Systems” node, which was created to capture those articles that explicitly identified systems theory and related approaches including learning organizations. As noted earlier, there are elements of conceptual consistency or continuity between cultural theories and systems theories. However, the appearance of systems theories as a frame used in a subset of studies was not anticipated. That appearance, however, provided an opportunity to look more closely at this subset of evaluations in an effort to assess whether those studies provided conceptual or operational evidence that would support the existence or understanding of the relationship between safety culture and the safe systems approach. A closer review of these evaluations did reveal a small number of references that have a shared intellectual genealogy with the social-technical systems theory that informs the safe systems approach, but no direct or shared sources between the two bodies of thought were identified. Although no direct link was found, the appearance of these concepts in culture-based interventions, coupled with the shared intellectual genealogy and conceptual continuity suggest that there may be more robust intersections or synergies between the two to be further explored and developed.

Figure 3 is the node tree that identifies the Outcome Indicators or Measures that were revealed in the analysis of the literature. As was expected, the most common outcome indicators and measures reflected common attributes of most cultural theories, namely that there are particular attitudes/beliefs, artifacts, or structures (e.g., policies, procedures and processes, as well as formal or informal structural forms such as authority or patterns of communication). It’s also notable that a substantial number of articles explicitly recognized that particular values, like caring or transparency, were important indicators or measures of culture and culture change. These attributes are, in turn, often linked to culture-based behaviors, or behaviors that express or reflect the attitudes and values of the culture. Not surprisingly, the analysis also identified that organizational performance outcomes, such as patient or student outcomes, were also present in the literature.

FIGURE 3

FIGURE 3. Outcome indicator-measure node tree map.

In addition to looking at the basic trends in the literature related to the cultural and other theories that inform the interventions and strategies and the outcome indicators and measures used, our analysis also examined the intersection or overlap between different concepts and their codes.²

The first of these queries examined the intersections between the cultural or other theory used to inform the intervention and study and the outcome indicator or measure used. Several elements of this query, the results of which are shown in Table 1, are notable. The first is that indicators and measures that are consistently a part of cultural theories, namely attitudes, beliefs, and behaviors are by far the most common indicators and measures used in the studies we compiled, especially among those articles that work either from a cultural model based in Edgar Schein’s work or that have some other, specific cultural framework. Similarly, a substantial number of these same articles identify that culture is expressed or embedded in the settings studied, into artifacts, namely things like policy, procedure, and practice or in formal or informal structures such as authority or communications.

TABLE 1

TABLE 1. Matrix query results: Culture by outcomes indicators/measures.

Another notable pattern has to do with the commonality of “values” as an indicator in the studies in this sample. Nearly all cultural theories recognize that any given culture is likely to include a fairly specific set of shared values. However, relatively few studies, regardless of the underlying cultural theory identified, have values as an indicator or variable to be changed or augmented as a result of the intervention. One caveat to this comment is that values can overlap with beliefs and particularly attitudes. For example, a safety culture that prompts workers to care for each other such that they are more likely to intervene to prevent risky behavior could describe care as a behavior, an attitude, or a value.

Lastly, and somewhat unexpectedly, relatively few studies identified or focused on outcome measures that a culture would purportedly influence (i.e. patient outcomes, student outcomes, or accidents/collisions). Even among those studies of healthcare organizations, which as we will describe further below, were by far the most common industry present in the sample of articles we identified, relatively few studies focused on those ultimate, program outcomes. The vast majority of studies focused primarily on intervening variables, which are components or elements of culture, rather than outcome variables that culture is intended to impact. One other indicator identified through the open coding process, was compliance–i.e., whether the organization, unit, or individuals studied complied with applicable regulatory regimes. In this way, compliance serves as both a convenient proxy for other outcomes, like patient outcomes; it may also be an outcome in and of itself, in that compliance, especially if non-compliance results in sanctions, can easily become an outcome.

A second matrix query, shown in Table 2, focused on cultural theories within this literature in relation to any outcome effects identified in the study. Column one (Outcome Effects) includes those studies that identify interventions had positive effects, column two (Outcome - Mixed) includes those studies with mixed or some combination of both positive and negative effects, or no effects were found or described for those studies in column three (Outcome-No Effect). As Table 2 indicates, the vast majority of the studies in our sample found mixed or positive effects. Within studies associated with particular cultural or other organizational theories, those that fell within “Other” node had the largest portion of studies that had mixed and especially positive outcomes. Those studies that didn’t articulate or specify an explicit model of culture had the fewest number of studies with positive or mixed results. While it would require more research to identify any causal link between causal theory and program outcomes, these findings suggest the possibility that a lack of a clear cultural theory may result in the lack of a clear causal model, either to establish a culture change intervention or to establish a model for how culture impacts the performance outcomes of the organization. In the absence of a clear causal model, it may be more difficult to craft a program that effectively impacts either culture or organizational performance outcomes.

TABLE 2

TABLE 2. Matrix query results: Culture by outcome.

The final query centered on cultural theories, in this case in relation to the unit of analysis used by the studies in the sample. The results of this query, which are shown in Table 3, indicate that a large majority of the studies in the sample, regardless of the underlying cultural or organizational theory, focus on interventions that target the organization as a whole or a specific unit within the organization. A smaller, but still substantial number of studies focused on change industry-wide. Only a small portion of the studies focused on change primarily or solely at the level of the individual.

TABLE 3

TABLE 3. Matrix query results: Culture by unit of analysis.

Industries and Sectors

The second area of inquiry and analysis focused on patterns associated with the various industries or sectors of society within which the articles in the sample fell. The node tree in Figure 4 shows the seven sectors or industries from which nearly all of the studies in our sample fell. As we noted in the methods section above, the search strategy used to identify articles to be included in this analysis did seek to find traffic safety culture articles but was otherwise neutral with respect to targeting industries or sectors. The search terms and strategy focused on organizational culture, safety culture, and culture change, regardless of setting.

FIGURE 4

FIGURE 4. Industry node tree map.

By far the largest number of articles were focused on the healthcare industry and healthcare organizations. Education and transportation accounted for a substantially smaller, but still notable, portion of the studies in the sample. Studies focused on organizations in the energy industry and those that we classified as “Private Industry” saw slightly fewer studies than education and transportation. Private Industry, for the purposes of coding articles in this study, included any private sector organization or operation outside of energy or transportation. The final two sectors, Economic Development and Community Development, were created as axial codes, meaning that we anticipated that these would be common areas of focus in the literature, but that was not the case. Our sample of articles included only one article in each of these two areas.

In addition to examining the commonality of different industries in the literature, we also looked for patterns in the relationship between the industries and settings in the sample and other variables or parent nodes in the overarching coding scheme. The first relationship we looked at was between industries or settings and the theories of culture used as a part of the studies and interventions. Table 4 shows the results of this matrix query and indicates that the majority of studies across industries have adopted and articulated some theory of culture. Interestingly, the majority of studies that were coded into the “No Clear or Explicit Model” node came from the healthcare industry. While this would appear to partly reflect the large number of studies from that industry, this result would seem to merit further investigation, particularly in line with the question posed in the previous section about whether the lack of a clear cultural theory leads or is related to the lack of a clear causal model between change initiatives and culture or between culture or organizational performance outcomes. The results of this matrix query also shed some light on the presence and distribution of learning-systems theory models across the industries in our sample. Again, the presence of learning and systems theories is interesting because of the notional continuity between systems theories, the concept of culture, and emergent safe-systems approaches in traffic safety. As can be seen in Table 4, the learning-systems models are fairly broadly distributed across the industries in our sample, and do not appear to be especially prevalent in any one industry.

TABLE 4

TABLE 4. Matrix query results: Industry and culture model.

One further analysis looked at the relationship between industry and other nodes or variables and sought to identify patterns between industry and the outcome indicators or measures used by the studies in each sector (see Table 5). As was true of the examination of the intersection of culture theories and indicators and measures used to evaluate those interventions, this analysis reveals a general pattern of culture-based attributes including attitudes, behaviors, and artifacts present across industries. There were some exceptions to this general pattern. For example, none of the studies from private industry focused on behaviors. As was noted earlier, few studies in the sample explicitly identified values as an indicator within their studies, though the analysis by industry suggests that those initiatives and studies that did identify value indicators were in the healthcare and education sectors respectively.

TABLE 5

TABLE 5. Matrix query results: Industry and outcome indicator/measure.

The final analysis conducted with industry or setting being a key focus looked at patterns in the relationship between industry and outcome effects, again coded as positive outcome effects, mixed effects, or no effects (Table 6). As was the case with the results of the matrix query above looking at the relationship and patterns between culture models and outcome effects, here again we found that the majority of studies across industries found positive, or at the least, mixed outcomes resulting from the interventions evaluated in each article. There were a small number of studies from the healthcare industry that were unable to identify positive effects. Whether the presence of these few negative evaluations is a feature of a tendency toward more robust or critical analysis in the healthcare industry, a feature of the larger number of studies being more likely to have some negative outcomes, the lack of a clear cultural theory or corresponding causal model, or some other or combination of these factors is not clear from these data. However, the pattern of results suggests that culture change interventions and interventions using culture to impact other outcome variables can have positive impacts on culture and ultimately on other organizational performance or outcome measures.

TABLE 6

TABLE 6. Matrix query results: Industry and outcome effects.

Research Designs and Implications

The last category of analysis looked at the research designs identified in the articles in our sample generally and within each of the industries or sectors in the sample. Broadly, Figure 5 indicates that, across the entire sample of articles analyzed, qualitative, quasi-experimental,³ and single-group mixed methods designs are the most frequently used to study specific initiatives or interventions. Typically, mixed methods designs use a combination of quantitative and qualitative approaches, though in some instances studies use a mix of different quantitative designs (e.g. surveys and secondary performance data). The analysis also found a smaller but still substantial number of systematic analyses, meta-analyses, or literature reviews, all of which drew together and examined the existing literature in various ways. In addition to these designs, the analysis identified a small number of single-group, pretest/post-test studies and formative evaluations. Although they tend to be logistically and programmatically difficult to conduct, the analysis did identify a small number of studies self-described as experimental.

FIGURE 5

FIGURE 5. Design node tree map.

When examining the pattern of designs in relationship to industry, several observations can be made from the results of the matrix query summarized in Table 7. First, it is striking that two of the most common designs within healthcare are quasi-experimental and qualitative. While these two approaches are sometimes considered quite different in terms of their aims and even their rigor, on further consideration it seems reasonable that these distinct strategies are relatively common because they provide different forms of information about organizational performance and thereby enable healthcare organizations to present different information or make different kinds of arguments to distinct audiences. Other notable elements of this query include a larger portion of single-group, mixed methods approaches in education. Although the relative number of studies drawn from the transportation sector is relatively low, it is also notable that those studies fall into just three categories: meta-analysis, single-group mixed methods, and systematic analysis. Although distinct, if the meta-analysis studies and systematic analyses are combined based on the logic that both approaches collect, aggregate, and assess data from across existing studies, then these aggregating approaches are disproportionally common in the transportation sector. This may reflect the fact that the transportation industry has come to utilize culture-based approaches after other sectors have already done so and, as a result, turn to aggregating designs as a way of taking a broader look at what is known to more quickly assimilate relevant learnings into efforts in that industry.

TABLE 7

TABLE 7. Matrix query results: Industry and research design.

Discussion and Conclusions

Implications of the Current Study

The patterns and relationships identified through the examination of this literature lead to several observations.

1. There is wide-spread evidence across the literature indicating that initiatives to influence organizational culture or safety culture more narrowly, or to leverage culture as a means of improving other performance outcomes can be effective. This appears to be true regardless of the specific cultural theory utilized to inform the intervention. There are, however, a surprising number of studies that lacked a clear model or definition of culture. The absence of such a model raises the possibility that there may be a corresponding lack of clarity in the causal models assumed between the interventions used and anticipated culture impacts or between the cultural attributes and their impact on performance outcomes. This, in turn, makes both formative and summative evaluations of the program more challenging as well, despite strong arguments in the literature for the value of doing so (Hoole and Tracy Patterson 2008; Government Accountability Office 2013; Holosko 1996). While the rationale for ensuring the presence and use of clear and precise cultural models seems important for program development and eventual evaluation, additional research would nevertheless be necessary to further disentangle and assess these relationships. Specifically, there is a need for further research to examine questions like how important it is to have a clear theoretical grounding, or whether any particular intervention is likely to be as successful as the next, regardless of whether there is any clear model either articulated by the program staff or embedded by reference by program designers who draw from those ideas only to have them become obscured. Regardless, having a clear definition and model of culture, and a correspondingly clear causal model that links either elements of an intervention to cultural attributes in the target organization or setting, or one that clearly links culture features to expected outcome variables would seem to be a benefit to both scholars and practitioners.

2. The appearance of systems-oriented frames among the cultural theories used for the interventions in our sample, although surprising, is intriguing. At one level, the presence of system-oriented theory suggests the potential to integrate and leverage possible synergies between safety culture and the safe systems approach (Cox and Cox 1996; Larsson et al., 2010; Abel et al., 2020). The appearance of these ideas in our findings, coupled with their conceptual continuity and shared intellectual genealogy seems promising, especially given the prevalence of studies from the transportation, energy and even healthcare industries, where safety and health are core concerns. Given the small number of studies we found in traffic safety culture specifically, any insights drawn from other sectors where the intersections between safety culture and safe systems, especially human-machine interactions within those systems, would be valuable. Unfortunately, when looking at the presence of learning/systems theories across different industries, we did not find these approaches to be particularly common in industries like transportation and energy, where culture, especially OSC, is a more established. The limited presence of the connections indicates that there is need to flesh out and extend the conceptual links between systems thinking and organizational culture in a way that more fully conceptually links them, and to do so with enough operationalizable precision such that those links can be built into future interventions, and with enough clarity to be rigorously tested upon implementation.

3. Across the breadth of studies on our sample, we found a substantial amount of diversity of evaluation designs used to assess the impact of the respective interventions, including the indicators and measures of both inputs or independent variables and outputs and dependent variables. One consequence is that this diversity makes the possibility of generalizing any results or findings difficult, especially from the perspective of academically oriented, applied research and knowledge building. Practically, however, the breadth of designs is also reflective of the diversity of settings, intervention strategies and objectives sought across the initiatives within our sample. For practitioners, this reinforces the importance of selecting evaluation designs pragmatically, and in a way that is responsive to the questions sponsoring organizations and program managers need to answer, and reflective of the resources and capacity of the organization, rather than defaulting to perceived “ideal type” designs.

4. Among those studies that do have clearer theoretical grounding and corresponding clarity and specificity about critical variables and the relationship between them, there is a substantial amount of consistency in the indicators and measures across studies, industries, and even designs. Cultural attributes and expressions in the form of attitudes and beliefs, behaviors, and values, as well as the degree to which these attributes are manifested in the structures and practices of the organization–its artifacts–is widespread in this empirical literature. It is unsurprising that specific industries or sectors also develop additional outcome measures relevant to operations in that sector, whether related to compliance requirements that serve as a proxy for target outcomes, or measures of the target outcomes themselves. Despite the particularities of each context, the presence and specificity of the attributes and associated artifacts provides an opportunity for practitioners in other fields, in this case traffic safety culture, to identify, consider and potentially adopt or adapt cultural variables, artifacts/indicators and measures that have proven valuable elsewhere. Reflectively drawing on the learnings in other domains, can be a useful strategy to accelerate the development and use of effective practices, though with the important caveat that it’s critical for program practitioners to be attuned to the existing cultural context and its manifestation in things like the values and language of that setting. Moreover, as practitioners and researchers continue to conduct evaluations of culture-based change initiatives, it may be useful for evaluators and program managers to look beyond their industries as a way to identify new and evolving understandings of culture and its function, as well as the development of evaluation designs and research strategies.

Additional Recommendations for Future Research

As suggested in the discussion of the study limitations above, one recommendation for further research in this area is to broaden the search for published evaluations in order to identify and examine evaluations that appear in professional publications or that are self-published. Although the diversity of these publications and lack of a centralized search tool make the collection of these evaluations more difficult, the examination of evaluations beyond those in the peer reviewed literature has the potential to substantially expand the sample size. If this broader set of sources could be systematically gathered an analyzed, they could provide more insight into the models of culture being used, indicators and measures of culture, evaluation designs used, and outcomes identified.

A second line of research that will likely prove valuable is an extended qualitative analysis of the interventions used to change an existing culture, or to leverage culture in support of improving safety behavior. Because the structure and function of culture is highly contextual, a qualitative study of culture-change interventions can help to build a deeper, more nuanced and detailed understanding of the intervention strategies used, contextual factors at play, and the impacts of these various factors. This line of research will require shifting the unit of analysis from the article or report to the reference, in order to better identify and map themes and patterns related to the interventions and their impacts.

Finally, as the number of evaluations of culture-based interventions in traffic safety grows, it will be important to conduct further systematic, or even meta-analyses of these efforts. This will be particularly true for interventions that strive to change the culture of communities of road users rather than organizations.

Study Limitations

One methodological aspect of the approach used in this study that should be noted again here is that the coding process used was primarily focused on the item–i.e., the article–as the unit of analysis rather than the number of references coded within each article. The focus on coding items rather than all references within each item allowed us to discern patterns across the body of literature that might have been obscured if we had focused on coding every reference in every article. However, this coding strategy does not result in a set of codes that are entirely mutually exclusive. For instance, when coding for the cultural theory used in each study, it is likely that a study falls into only one code category, meaning that it uses only one cultural theory to inform the intervention and evaluation. However, if a study identified two distinct theories, both would be coded and both will appear in the results of relevant queries done for the analysis. As a result, when presenting the results of various matrix queries conducted in the next section, we have chosen not to report the numbers of items generated by the query but instead have presented them in color-coded/shaded categories that represent a range of frequencies. This approach is consistent with the norms of reporting the findings of qualitative research, and because unlike cross-tabs from a quantitative design, the row and column totals from a matrix query may vary slightly from the total number of items in the sample, potentially causing confusion.

A second issue is that despite the efforts used to broaden the scope of this study to include culture interventions in other industries and towards ends beyond safety, the study is still limited to a fairly small sample of evaluations, in part because of the decision to utilize published, peer-reviewed articles as the unit of analysis. This was a strategic choice in that it the peer-review process and journals where research is published provided a greater degree of consistency and continuity with respect to elements including methodologies, terminologies and grounding in shared literatures. This choice did, however, have the likely effect of reducing the overall sample size of material for the study, by comparison to including studies published outside the peer-reviewed literature. As a result, it is difficult to draw strong conclusions about the trends and patterns, and their implications for practice, particularly within any one industry or program area.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Materials, further inquiries can be directed to the corresponding author.

Author Contributions

All authors contributed equally to the conceptualization of the study, development of research questions and research design. EA, NW, JO, and KG collaborated on the interpretation of the data, discussion and implications. HW provided research support including data collection and management, and for coding and conducting preliminary analysis of the data.

Funding

This research was funded by the Montana Department of Transportation and the Federal Highway Administration. This research has concluded and Montana Department of Transportation has accepted the final report.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

¹It should be noted that coding for this study focused on items or files, rather than references. For example, when coding for the research design used, our concern was identifying the research design identified in each article, rather than the number of times each article referred to the research design used. The focus on items or files rather than references within each file allowed us to treat the article as the unit of analysis rather than the concept. Qualitative studies that use this form of content analysis are often more interested in the frequency of references, for example the number of times interviewees mention a topic, because it reveals something about the participants’ concerns, perceptions and priorities as indicated by frequency of reference, which can then be tranced into how those concerns appear across the sample of participants as a whole. However, because our interest is on the patterns within the literature, a focus on references risks obscuring those patterns.

²Matrix Queries in NVivo are similar to cross-tabs used in other quantitative studies. Because of the qualitative approach used here, the output of the matrix queries is presented graphically and in narrative form, rather than using frequencies.

³For the purpose of this study, quasi-experimental designs included only those that utilized a comparison group, and did not include time-series studies as described by Posavac (2011). By comparison, experimental designs also use a comparison group, but randomized assignment into the comparison and experimental groups, and also use a single or double-blind strategy for participants and/or researchers.

References

Abel, S, Lindley, J. A., and Paniati, J. F. (2020). The Road to Zero: Taking a Safe System Approach. ITE J. 90, 26.

Google Scholar

Brescianai, M. J. (2011). Identifying Barriers in Implementing Outcomes-Based Assessments Program Reivew: A Grounded Theory Analysis. Res. Pract. Assess. 6, 5–16.

Google Scholar

Clarke, S. (1999). Perceptions of Organizational Safety: Implications for the Development of Safety Culture. J. Organiz. Behav. 20, 185–198. doi:10.1002/(sici)1099-1379(199903)20:2<185::aid-job892>3.0.co;2-c

CrossRef Full Text | Google Scholar

Cox, S., and Flin, R. (1998). Safety Culture: Philosopher's Stone or Man of Straw? Work & Stress 12, 189–201. doi:10.1080/02678379808256861

CrossRef Full Text | Google Scholar

Cox, S. J., and Cox, T. (1996). Safety, Systems and People. Oxford: Butterworth-Heinemann.

Google Scholar

Edwards, J. R. D., James, F. D. S., and Watson, B. (2014). A Framework for Conceptualising Traffic Safety Culture. Transp. Res. Part F Traffic Psychol. Behav. 26, 317–325. doi:10.1016/j.trf.2014.03.002

CrossRef Full Text | Google Scholar

Edwards, J. R. D., Davey, J., and Armstrong., K. (2013). Returning to the Roots of Culture: A Review and Re-conceptualisation of Safety Culture. Saf. Sci. 55 (June), 70–80. doi:10.1016/j.ssci.2013.01.004

CrossRef Full Text | Google Scholar

Geertz, C. (1973). The Interpretation of Culture. New York: Basic Books.

Google Scholar

Government Accountability Office (2013). Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making. Washington, DC: General Accountability Office.

Google Scholar

Gulick, L., and Urwick, L. (1937). Notes on the Theory of Organization. New York: Columbia University, Institute of Public Administration.

Google Scholar

Hale, A. R. (2000). Culture’s Confusions. Saf. Sci. 34 (1–3), 1–14. doi:10.1016/s0925-7535(00)00003-5

CrossRef Full Text | Google Scholar

Harrison, S. H., and Corley, K. G. (2011). Clean Climbing, Carabiners, and Cultural Cultivation: Developing an Open-Systems Perspective of Culture. Organ. Sci. 22 (2), 391–412. doi:10.1287/orsc.1100.0538

CrossRef Full Text | Google Scholar

Håvold, J. I., and Nesset, E. (2009). From Safety Culture to Safety Orientation: Validation and Simplification of a Safety Orientation Scale Using a Sample of Seafarers Working for Norwegian Ship Owners. Saf. Sci. 47 (3), 305–326. doi:10.1016/j.ssci.2008.05.002

CrossRef Full Text | Google Scholar

Hedlund, J. (2007). Improving Traffic Safety Culture in the United States: The Journey Forward Summary and Synthesis. Washington, D.C., US: AAA, Foundation for Traffic Safety.

Google Scholar

Hoekstra, T., and Wegman, F. (2011). Improving the Effectiveness of Road Safety Campaigns: Current and New Practices. IATSS Res. 34 (2), 80–86. doi:10.1016/j.iatssr.2011.01.003

CrossRef Full Text | Google Scholar

Holosko, M. J. (1996). Obstacles to Conducting Program Evaluations:. J. Health & Soc. Policy 8 (1), 91–101. doi:10.1300/J045v08n01_08

CrossRef Full Text | Google Scholar

Hoole, E., and Patterson, T. E. (2008). Voices from the Field: Evaluation as Part of a Learning Culture. New Dir. Eval. 2008, 93–113. doi:10.1002/ev.270

CrossRef Full Text | Google Scholar

Kapucu, N., Hu, Q., and Khosa, S. (2017). The State of Network Research in Public Administration. Adm. Soc. 49 (8), 1087–1120. doi:10.1177/0095399714555752

CrossRef Full Text | Google Scholar

Katz, D., and Kahn, R. L. (1978). The Social Psychology of Organizations. New York: Wiley.

Google Scholar

Larsson, P., Dekker, S. W. A., and Tingvall, C. (2010). The Need for a Systems Theory Approach to Road Safety. Saf. Sci. 48 (9), 1167–1174. doi:10.1016/j.ssci.2009.10.006

CrossRef Full Text | Google Scholar

Von Bertalanffy, L. (1950). An Outline of General System Theory. Br. J. Philosophy Sci. 1, 114–129. doi:10.1093/bjps/i.2.134

CrossRef Full Text | Google Scholar

Luria, G., and Rafaeli, A. 2008. “Testing Safety Commitment in Organizations through Interpretations of Safety Artifacts.” J. Saf. Res. 39: 519–528. doi:10.1016/j.jsr.2008.08.004

CrossRef Full Text | Google Scholar

Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 6 (7), e1000097. doi:10.1371/journal.pmed.1000097

PubMed Abstract | CrossRef Full Text | Google Scholar

Nieva, V. F. (2003). Safety Culture Assessment: A Tool for Improving Patient Safety in Healthcare Organizations. Qual. Saf. Health Care 12, 17ii–23. doi:10.1136/qhc.12.suppl_2.ii17

CrossRef Full Text | Google Scholar

Otto, J., Ward, N. J., and Finley, K. (2019). “Guidance for the Measurement and Analysis of Traffic Safety Culture,” in Traffic Safety Culture: Definition, Foundation, and Application (Bingley, UK: Emerald Group Publishing Limited), 65–91. doi:10.1108/978-1-78714-617-420191006

CrossRef Full Text | Google Scholar

Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. New York: Basic Books.

Google Scholar

Rasmussen, J. (1997). Risk Management in a Dynamic Society: A Modelling Problem. Saf. Sci. 27 (2–3), 183–213. doi:10.1016/S0925-7535(97)00052-0

CrossRef Full Text | Google Scholar

Reason, J. (2000). Safety Paradoxes and Safety Culture. Inj. Control Saf. Promot. 7, 3–14. doi:10.1076/1566-0974(200003)7:1;1-v;ft003

CrossRef Full Text | Google Scholar

Robertson, R. D., and Pashley, C. (2015). Road Safety Campaigns: What the Research Tells Us. Ottawa, Canada: Canadian Automobile Association, Traffic Injury Research Foundation of Canada.

Google Scholar

Schein, E. H. (2004). Organizational Culture and Leadership. 3rd ed. San Francisco: Jossey-Bass.

Google Scholar

Senge, P. M. (1990). The Fifth Discipline: The Art and Practice of the Learning Organization. New York: Doubleday.

Google Scholar

Simon, H. A. (1976). Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. New York: Free Press.

Google Scholar

Strauss, A. L., and Corbin, J. M. (1990). Basics of Qualitative Research : Grounded Theory Procedures and Techniques. Newbury Park, CA: SAGE.

Google Scholar

Ham-Baloyi, W., Jordan, P., and Jordan, P (2016). Systematic Review as a Research Method in Post-graduate Nursing Education. Health SA Gesondheid 21, 120–128. doi:10.1016/j.hsag.2015.08.002

CrossRef Full Text | Google Scholar

Turnbull, K. (2011). Improving Roadway Safety Programs through University–Agency Partnerships. Washington, D.C., US: Keck Center of the National Academies.

Google Scholar

Walton, M. (1986). The Deming Management Method. New York: Perigee.

Google Scholar

Ward, N. J., Otto, J., and Finley, K. (2019). “Ten Principles of Traffic Safety Culture,” in Traffic Safety Culture: Definition, Foundation and Application (Bingley, UK: Emerald Group Publishing Limited), 21–40. doi:10.1108/978-1-78714-617-420191004

CrossRef Full Text | Google Scholar