How can educational innovations become sustainable? A review of the empirical literature

Abstract

In literature there is a great variety in the definitions and identified critical features of sustainability of educational innovations. The aim of this review is to provide an overview of the entire range of factors influencing the sustainability and its core aspects. A systematic review was performed in which electronic databases were searched for peer reviewed articles, published between 2002 and 2017. Based on results of 44 publications, the following definition could be constructed: “Sustainability refers to the process of integrating the intervention's core aspects in organizational routines, which are adaptive to ongoing work, with maintenance or continuation of improved results.” We found four main factors influencing sustainability of educational innovations: school organizational, innovation, individual, and context characteristics. The empirical-based model developed in this review should be validated in practice to create transparency and focus in sustainability research.

Introduction

Schools are faced with the challenge of implementing new practices; however, many innovations that are initially successful fail to become part of a school's organizational routines (Wiltsey Stirman et al., 2012). The hardest part of any educational innovation is not how to start, but how to sustain the innovation within the organization (Hargreaves and Fink, 2012). Moreover, sustainability is not a matter of black and white, but nuanced (Tappel et al., In Press).

Educational innovations appear to follow multiple phases to ultimately accomplish a change in education, which can be described as initiation, implementation (e.g., Fullan, 2007; West, 2012) and continuation/sustainability (Fullan, 2007). The different phases are interdependent, and later phases are influenced by decisions made in the earlier phase(s) (Ertesvåg and Vaaland, 2007; Fullan, 2007). Sustainability within education is defined in numerous ways, in which core aspects of sustainability differ across definitions. Coburn and Turner (2012), for example, emphasized the aspect of the visibility of the innovation through organizational routines. An organizational routine is a pattern within an organization of recurring actions that influence each other and require the involvement of multiple actors, in which a distinction can be made between ostensive and performative aspects (Feldman and Pentland, 2003). The ostensive aspect is defined as the perception or structure of the routine. The performative aspect is defined as the specific actions that are undertaken to perform the organizational routine. Once implemented, such routines support the initiation of coordinated actions between individuals (Spillane, 2012).

Another example of an aspect of sustainability often described in the literature is that the innovation can be operative during regular work without causing interruption of existing practices (Coburn and Turner, 2012) and should be integrated with other existing initiatives within the organization (Hargreaves and Fink, 2008). They additionally referred to sustainability as an ongoing innovation process after removal of support. Other definitions do not include the aspect of not being disruptive of existing practices or processes (Copland, 2003; Fullan, 2005; Coburn et al., 2012). Overall, there seems to be a lack of uniformity as far as exactly what sustainability is, and thus also with regard to how it can be measured to determine the effects of educational innovations over the longer term.

What is also known from the literature is that sustainability is influenced by a multitude of factors. Understanding the factors and processes involved in sustainability is at least as important as the implementation of an innovation itself (Wiltsey Stirman et al., 2012), if we want to realize sustainable school improvement.

Although an extensive body of literature is available on the sustainability of educational innovations, there is at the same time a great variety in the definitions and factors influencing sustainability. Much of this literature has a theoretical focus, and does not provide clarity about its empirical foundations. Influential factors also vary between different articles. A general model of the sustainability of educational innovations seems to be lacking thus far. Therefore, there is a need for in-depth analyses and evidence-based theories that explain the long-term success or failure of educational innovations (Cohen and Mehta, 2017). The aim of the systematic review is to create more clarity on what sustainability of educational innovation is. Moreover, as educators face the problem of sustainability in their organizations, we also aim to provide an overview of the entire range of factors influencing the sustainability of educational innovations. The research questions are:

• How can sustainability of educational innovations be defined in terms of its core characteristics?

• What factors influence the sustainability of educational innovations?

Methods

This study used a stepwise process for conducting a systematic review in the social sciences (Petticrew and Roberts, 2006). This process consisted of: (1) defining the research question; (2) defining the search terms; (3) choosing literature databases; (4) conducting the literature search; (5) formulating inclusion criteria; (6) selecting literature, using the inclusion criteria; (7) data extraction; and (8) aggregation and synthesis of the evidence. A library professional was consulted to advise on our literature search. A data extraction form was used for each relevant publication, to collect the information needed to answer the research question. Additionally, a scientific quality check of each publication ensured that only studies that met the data extraction quality criteria were selected for analysis (Petticrew and Roberts, 2006). Figure 1 shows the quorum flowchart for the selection process.

Figure 1

Data extraction

A data-extraction form was designed for analyzing the selected publications, based on the data-extraction form used by Hoogland et al. (2016). The use of a form ensured that comparable data could be gathered from the selected publications (Petticrew and Roberts, 2006).

Additionally, the data extraction form included 12 questions regarding the quality of the research.

Results

A total of 44 publications were found suitable for data extraction and analysis. The characteristics of the included studies are shown in Table 1.

The educational contexts varied considerably between studies found in this review. The studies focused on a broad scope of innovations founded in diverse countries. This wide distribution around the world indicates that this topic is on the radar in many parts of the world. Over 40% of the studies took place in the USA compared to 32% in Europe. This should be noted when interpreting the results. The results included literature on specific innovations (e.g., experimental schools), interventions (e.g., positive behavior support), reforms (e.g., school reforms) and partnerships (e.g., involvement university in schools). These will all hereafter be referred to as “innovations,” as they all involved “the intentional introduction and application within a role, (work)group, or organization of ideas, processes, products or procedures new to the relevant unit of adoption, designed to significantly benefit the individual, the group, the organization or wider society” (West and Farr, 1990, p. 9). Most of the studies were qualitative case reports (24), 16 cross sectional studies were included and four quasi-experimental studies.

Sustainability

Most of the articles, 31 out of 44, reported no explicit definition of sustainability, but did mention important aspects of sustainability. The most frequently addressed specific (core) elements of sustainability in these papers are: (1) results continued to improve/results maintained; (2) sustained implementation of (the core elements of) the innovation; (3) integral part of daily school routines; and (4) adaptation over time. Table 2 gives an overview of articles which mentioned these elements.

Results continued to improve/maintained

Of the 44 articles, 14 described a core element of sustainability as maintaining or continuing improvement of results following an innovation. This element refers to the need of innovation's results to keep being visible or being improved over time. All studies measured the results of the innovations multiple years later in order to determine sustainability of results. For example, Alanís and Rodríguez (2008) measured in their case report study sustainability as students consistently outscoring their peers in the district and state for multiple years on English and mathematics achievement scores. Stringfield et al. (2008) described sustainability in their cross-sectional study as schools keep making strong academic progress 5 years after an innovation. All studies measured the results of the innovations multiple years later in order to determine sustainability of results.

Sustained implementation (of the core elements) of the innovation

Thirty four articles referred to the sustained implementation of the innovation as to continue with the core elements of the innovation over time, despite adjustments that possibly have to be made to integrate an innovation into the organization. Almost half of these 34 articles reported that an implementation is successfully sustained when the core aspects of the innovation are implemented over the long term. Core components refer to components of the intervention long term goal of the intervention. Core components of an intervention include the functions or principles and related activities necessary to achieve outcomes (Blase and Fixsen, 2013).

The extent to which these core aspects were specified differed across articles. Multiple articles stated that the core aspects or essential form should be adhered to (e.g., Andreou et al., 2015; Bean et al., 2015), or referred to a standard of practice (e.g., Elias, 2010), to fidelity to core program principles in a piecemeal manner (e.g., Mathews et al., 2014), to continuity of issues (Roffe, 2010), or to quality of partnership program over time (Van Voorhis and Sheldon, 2004).

Integral part of organizational routines

Nine studies stated that innovations are sustainable if they have become a regular part of (organizational) routines within the school or district. This means that school leaders and teachers do not perceive the innovation as something new or added to their practice, but as something that has become a routine part of their practice and they have to be involved in.

Adaptation over time

This element refers to the adaptability of an innovation to the organizational routines within an organization, but at the same time adhering to the core elements of an innovation. Three case report studies described the core element of a sustainable innovation as being adaptive over time (Benz et al., 2004; Deaney and Hennessy, 2007; Elias, 2010). Benz et al. (2004) focused in their study on how the program developed and changed over time. Deaney and Hennessy (2007) discussed sustainability in light of “evolution over time.” They did not consider the sustainability of the initial innovation in a static form (often referred to as fidelity), but considered that sustainability is also about a development over time, but at the same time adhering to the core aspects of the innovation. Teachers in their study described how they had integrated the new practice into their departmental schemes of work. Deaney and Hennessy distinguished different mechanisms for accomplishing these changes: by means of trialing (experimenting to see what works) and by means of feedback from colleagues. Elias (2010) emphasized “the necessity for ongoing flexibility of practices to promote and reinforce the innovation, as opposed to the conceptualization of a set list of practices that may be prey to extinction if evolving school schedule, budget, or other requirements conflict with the practices as initially implemented” (p. 19).

Table 2 gives an overview of the aspects of sustainability and the corresponding numbers of the articles.

Influential factors

Multiple factors could be identified in the selected articles that influenced the sustainability of innovations. Many of these factors work both ways: their presence often fosters sustainability and their absence often hinders sustainability. In studies on educational innovations the importance of the following categories of factors is often discussed (e.g., Rikkerink, 2011; Coburn and Turner, 2012; Datnow et al., 2012; Hoogland et al., 2016): Characteristics of the school organization, leadership, individual (teacher) characteristics, and context. We used these categories to structure our findings. However, while analyzing our results we discovered a category which is often not mentioned in the literature: Characteristics of the innovation. Moreover, the type of leadership found to be important in the studies described in the review seem to move away from leadership as a personal characteristic, and describe leadership more as an organizational quality. This is also mentioned by Spillane et al. (2004). Leadership in this case is spread over several people (with formal and no formal leadership roles) in an organization (Spillane et al., 2008; Spillane, 2009). After several rounds of clustering, we concluded that overall four different categories of factors could be distinguished: characteristics of the school organization, of the innovation, of the individual, and of the context. Table 3 illustrates the division of factors into the different categories, including the frequency of the found factors and in which of the included studies.

School organizational characteristics

Collaboration

A total of 8 publications stressed the importance of some form of collaboration. Although we acknowledge the different forms and operationalizations of collaboration that exist in literature (e.g., Little, 2010), for this review we used the following definition: Collaboration means working together on the same shared problem and goals (Lawson, 2004). This takes place through talk, action, and reflection among individuals a community of learners emerges. Collaboration can create a space that enables us to challenge taken-for-granted ways of working together and to bring about transformation in educational practice (Goulet et al., 2003). A quasi-experimental study found that an innovation is less likely to be sustainable when the people involved work in isolation; educators in schools (teachers, support staff and school leaders) need to collaborate to sustain innovations. When teachers struggle collectively with complicated challenges concerning the implementation and sustainability of innovations, they feel empowered and more significant (Gilad-Hai and Somech, 2016). Also, the case report study of Zehetmeier (2015) found that it is important that teachers engage in joint reflection and communication. Multiple studies found that collaboration for sustainability is not restricted to the same team, grade or school (e.g., Bean et al., 2015; Pinkelman et al., 2015; Drits-Esser et al., 2017). References to the importance of collaboration within networks (between schools, districts, and communities) to support collaborative practice and to share the costs of, for example, expert support are often made (e.g., Benz et al., 2004; Edwards Groves and Rönnerman, 2013).

Knowledge sharing within and outside the organization

Knowledge sharing influenced the sustainability of innovations according to 16 articles and can be defined as an activity through which knowledge (information, skills, materials, or expertise) is exchanged among people, communities, or organizations (e.g., Bukowitz and Williams, 1999). Knowledge sharing can also be seen as a form of collaboration (e.g., Little, 1990).

Within the school organization, multiple studies found that staff should present their findings and the results of working with the innovation to each other (e.g., Stringfield et al., 2008; Lewin et al., 2009). The quasi-experimental study of Gaikhorst et al. (2017) found that staff should be given opportunities to do so in order to sustain an innovation. Dekker and Feijs (2005) reported in their case report study on the importance of considering how ideas “travelled” and that there should be plans to disseminate ideas from the innovation. Collegial relations and communication among staff have been found important preconditions for this (Kirtman, 2002; Elder and Prochnow, 2016). Personal contacts with colleagues in meetings and informal contacts have been found to enhance sustainability (Dekker and Feijs, 2005). Case report studies of Saito et al. (2012) and Andreou et al. (2015) found that building capacity within the school by sharing knowledge with new teachers enhanced sustainability.

For optimal knowledge sharing, it is essential that this is done effectively. In the cross-sectional study by Peters (2011), for example, participants valued knowledge sharing in the form of written materials about the program, and phone-calls and emails from the coordinators; accessibility of information was most important. Multiple studies found that an example of important information that should be shared within the school is the program's effectiveness: what is working and what needs adapting (Benz et al., 2004; Zehetmeier, 2015; Elder and Prochnow, 2016).

Outside the organization, networking has an important function. Exchanging ideas and information with other schools, sharing data, hearing how other school teams implement an innovation, and sharing concrete examples of practices were found to increase sustainability (Lewin et al., 2009; Andreou et al., 2015). Benz et al. (2004) found in their case report study that program effectiveness should be communicated within the network and explicit strategies for communicating these good results to school staff and administrators, parents, and community partners. Additionally, case report studies showed that integrated practices such as network monitoring, evaluation, planning, and a dissemination scheme with representation from all sectors and stakeholders are influential (Sanders, 2009; Roffe, 2010).

School culture

Overall, a supportive and open school culture was found important for the sustainability of innovations in eight publications and can be defined as the historically transmitted patterns of meaning that include the norms, values, beliefs, ceremonies, rituals, traditions, and myths understood, maybe in varying degrees, by members of the school community (Stolp and Smith, 1994). Several authors found that in order for an innovation to become sustainable, all members of the community should share a common understanding of the core components of the innovation (e.g., Coffey and Horner, 2012; Zehetmeier, 2015), which means, that there should be alignment between the program goals and the school's policy (Gaikhorst et al., 2017). The absence of a supportive school culture has been found to hinder sustainability. According to the findings of the case report study of Bambara et al. (2012) this implies a general lack of knowledge or awareness of activities, and long-held conflicting beliefs, values, and practices of school personnel. Payneeandy and Auckloo (2012) found in their case report study that a school culture where decisions are based on educators' intuitions and a culture of sticking to the textbooks hinders sustainability.

Support and feedback from colleagues

Support and feedback from colleagues (e.g., sometimes also seen as a form of collaboration and/or knowledge sharing, e.g., Little, 1990) influenced sustainability according to six publications. Gaikhorst et al. (2017) reported that support (such as sharing ideas and resources) and feedback from colleagues during the initial program stimulated sustainability of effects over the longer term. Through their involvement, colleagues and principals became aware of the themes that were discussed, and obtained insight into the kinds of expertise that the participants had developed. Based upon this knowledge, participants received opportunities to further develop their expertise after the program ended, as they were considered as experts on the subject by the principals and their colleagues (Gaikhorst et al., 2017). Sandholtz and Ringstaff (2016) reported that cross case patterns showed that teachers with ongoing collegial support, sharing ideas and resources were better able to sustain the instructional practices learned in professional development compared to those who did not received this. Collegiality has been found important here (Edwards Groves and Rönnerman, 2013).

Staff turnover

Low teacher and principal turnover is beneficial to sustainability, according to eight articles (e.g., Alanís and Rodríguez, 2008; Sandholtz and Ringstaff, 2016). Staff turnover can diminish staff knowledge and skills in daily practice, and can reduce staff commitment and consistency according to case report studies of Kirtman (2002) and Andreou et al. (2015). Moreover, they found that staff members who had been with the program a long time had a clearer understanding of their roles and responsibilities, the unique ways in which the innovation contributed to the school's overall services to students, and the importance of disseminating this knowledge to the larger school community. These members were also more effective in developing and maintaining purposeful relationships with other professionals in the school and community in order to support students and other staff (Andreou et al., 2015). Sandholtz and Ringstaff (2016) found that in schools that had significant turnover in principals, teachers found it challenging to adjust to changing instructional expectations.

Turnovers affected the stability of school policies, and consequently the sustainability of educational innovation (Saito et al., 2012).

Leadership

Numerous aspects of leadership were found to be important for sustainability. Multiple studies showed that principals are agents who can either help or hinder sustainability (e.g., Saito et al., 2012; Drits-Esser et al., 2017) or found that management and leadership are the most important and influential aspects for sustaining programs (Ng and Nicholas, 2013). Leadership can be provided by one or multiple persons. These are hereafter referred to as “leaders.”

Distributed leadership

Distributed leadership is a form of collective agency incorporating the activities of many individuals in a school who work at mobilizing and guiding other teachers in the process of instructional change (Spillane et al., 2004). Distributed leadership is about leadership both as a practice and as interactions, and is not restricted to those with formal leadership positions, but influence and agency are shared (Spillane and Diamond, 2007; Harris and DeFlaminis, 2016; Woods and Roberts, 2016). This includes everyone who contributes to leadership practices through influencing the motivation, knowledge, or practices of colleagues (Spillane, 2006; Harris and Spillane, 2008). Twelve articles pointed to the importance of distributed leadership with as argument that decisions should not be top-down, but should be made through a democratic process. It is therefore important to give teachers (but also parents and students, for example) ownership of and responsibility for the process (e.g., Kirtman, 2002; Jesson and Limbrick, 2014), and to collaborate closely with teachers (Payneeandy and Auckloo, 2012). They should be engaged in the decision-making process by means of shared leadership (e.g., Alanís and Rodríguez, 2008; Furman Shaharabani and Tal, 2017) and should be given local autonomy (Peters, 2011; Postholm, 2011). Teachers should therefore be educated to become teacher leaders (e.g., Elias, 2010), which is necessary for a sustainable educational future (Edwards Groves and Rönnerman, 2013). This means that organizational capacity for change has to be created by the leader to enable teacher leadership (King, 2016).

Vision, norms, and goals

Initiating and identifying a vision, norms and goals can be defined as a leader's role in contributing to building a shared vision, norms, and goals. This also includes setting priorities in the school (Moolenaar et al., 2012), and a more specific shared vision, norms, and goals for the innovation at hand. A leader's vision is another influential factor for sustainable innovations according to five articles which refers to the road to a mission, toward the goals of the organization (Fullan, 2006; Burk, 2013). The leader's demonstration of a strong philosophical stance (Bambara et al., 2012), and a continued and consistent focus has been found to influence sustainability (e.g., Bean et al., 2015). The case report study of Bambara et al. (2012) showed that this vision should be promoted, and should aim to reach sustainability. Martin et al. (2006) reported from their cross sectional study that this means that a vision needs to be formulated even before the implementation of the innovation, not only with regard to how to implement the innovation, but also with regard to how to sustain it. It is crucial that this vision is aligned with the staff's vision. A bottom-up approach to accomplishing this alignment was found to relate positively to sustainability of an innovation in a case report study (King, 2016). Moreover, the leader should be explicit to the staff about the priority of the innovation to the organization, which was found in six, mostly case report, publications (Larsen and Samdal, 2008; Bambara et al., 2012; Saito et al., 2012; Sanders, 2012; Andreou et al., 2015; Bean et al., 2015). Leadership buy-in has been found to be crucial (Bambara et al., 2012; Saito et al., 2012; Bean et al., 2015). For example, Andreou et al. (2015) described how an innovation within a district should be written into the district action plan and goals, with these goals set as a strong district priority; this ensures that the innovation is viewed as important by the schools. Larsen and Larsen and Samdal (2008) found that making the program a formalized strategy, an integral part of the school's activities and the school's policy, and making it mandatory for all teachers enhanced its sustainability. Lastly, Sanders (2012) described how the use of leadership strategies in prioritizing and mediating between individual and organizational factors, such as teacher alignment and commitment, formalization within policy, allocation of sufficient resources and maintenance of focus, are important for sustainability of innovations.

Providing individualized support

Providing individualized support can be defined as leaders who try to understand, recognize, and satisfy teachers' concerns and needs (e.g., by facilitating staff), whereas at the same time treating each teacher as an unique individual (Thoonen et al., 2011). This also involves actions, such as mentoring and coaching of teachers, delegating challenging tasks to teachers, providing feedback, and recognizing and talking to teachers about their needs and concerns (Thoonen et al., 2011). Leaders need to provide this type of support to teachers in order to sustain innovations, according to 17 of the articles with varying research designs. Studies found that leaders must be involved in the innovation in order to transfer expertise to the workplace (Payneeandy and Auckloo, 2012; Gaikhorst et al., 2017) and provide coaching and feedback to their staff (Larsen and Samdal, 2008; King, 2016) in order to enhance sustainability. It was found beneficial to sustainability if leaders showed appreciation for good practices, encouraged teachers to continue, and supported new ideas (e.g., Zehetmeier, 2015; Kafyulilo et al., 2016). Support enhances the interplay with teachers, and their motivation and enthusiasm. When leaders help to build capacity for change, and empower teachers to create collaborative learning cultures this increases sustainability (King, 2016). Studies showed that support from the leaders creates trust among the staff (Youngs and King, 2002), or can reduce the stress that is involved with educational innovations (Kirtman, 2002). This type of support also includes facilitation. Nineteen articles found that facilitation by the leader as far as providing time, money, and organizing resources enhanced sustainability, as it enabled the staff to carry out their work. In order to foster sustainability, funding needs to be appropriate and planned, and capacity has to be developed for the organization to assume some costs of the innovation without relying on external funding (e.g., Elias, 2010; Elder and Prochnow, 2016). Multiple studies also showed that facilitation needs to take the form of providing the relevant connections within appropriate networks (Lewin et al., 2009; Zehetmeier, 2015) and a safe working environment (Kafyulilo et al., 2016) in order to foster sustainability. Peters (2011) found that when issues around workload, space, planning and resource are not optimal, sustainability can be hindered.

Multiple studies found that access to external expertise or contact with a recognized researcher, consultant, or trained coach with outside information and tools, after the initial implementation phase, enhances sustainability of innovations (e.g., Mathews et al., 2014; Kafyulilo et al., 2016).

Knowledgeable and modeling leaders

Four case report studies found that it is important that leaders are knowledgeable about the innovation; they need to be well-informed about the program including procedural and conceptual knowledge of the new practice to enable sustainability (King, 2016) and, in addition, the knowledge to educate administrators about the innovation and the required transition (Benz et al., 2004). However, just being knowledgeable is not enough. Multiple studies with varying research designs reported that leaders also need to show that they are knowledgeable by engaging in modeling behavior and be actively involved in (district and school) training with regard to the innovation in order to change their own views if necessary, or strengthen their convictions in support of the innovation, for them to be able to propagate it (e.g., Bambara et al., 2012). They should share information about good practice (Lewin et al., 2009). By following up on the process of sustainability, the leader can model the value of the innovation's success to the organization (Larsen and Samdal, 2008). The leader must convey a positive “can do” attitude through both words and actions (Bambara et al., 2012), and has to be flexible, innovative and practical (Lewin et al., 2009). This modeling behavior enhances the respect a leader receives within the school, which adds to the sustainability of an innovation (e.g., Coffey and Horner, 2012; Sanders, 2012).

Communication

Effective communication between the leader and other key players within and outside the organization was found to be important in seven because it enhances engagement with the wider community (Benz et al., 2004; Ng and Nicholas, 2013). The leader should communicate his/her vision (Larsen and Samdal, 2008), and talk about the new practices (King, 2016). It is important to use local jargon when communicating with the different stakeholders (Payneeandy and Auckloo, 2012), for example, in interactions with parents (Ng and Nicholas, 2013). Unclear expectations were found to hinder sustainability (Peters, 2011), as did tensions between directives from above and wishes from below (Postholm, 2011).

Innovation characteristics

Effectiveness and efficiency

Characteristics of the innovation were found to have an impact on sustainability in eight, mostly cross-sectional, publications. Issues concerning the effectiveness, the support by evidence from research, and the efficiency of the innovation are crucial for sustainable implementation. Effectiveness refers to the realization of goals following an innovation, and efficiency refers to effectiveness in relation to the costs it has taken (e.g., in terms of time, work, etc.). In particular, these should be related to the expected benefits for the pupils and their enhanced achievement.

Structure

Twelve studies with varying research designs found that the extent to which an innovation is structured influences its sustainability in a positive way. Moreover, focused and long-term innovations are more likely to be sustainable (e.g., Gilad-Hai and Somech, 2016). These innovations should be used routinely as an integral part of the school organization (e.g., Ferguson et al., 2011). The continuity and consistency of the approach are therefore important (e.g., Pinkelman et al., 2015; Elder and Prochnow, 2016). Maintaining the core elements of an innovation while adapting the innovation to daily practice makes the innovation more efficient and effective (Andreou et al., 2015). It helps if an innovation fits in with other initiatives undertaken by the school board or government (Bean et al., 2015), with the curriculum (Peters, 2011), with the praxis orientation of the teachers (Edwards Groves and Rönnerman, 2013), and with the needs of the district (Sanders, 2012), and is integrated with other initiatives.

Built in positive reinforcement

Positive reinforcement was found to impact an innovation's sustainability in eight studies (case reports and cross sectional studies). This could, for example, be achieved by means of data use: by having observable and measurable information to track patterns of implementation and student outcomes, and the status and goals of their schools (e.g., Sanders, 2009; Pinkelman et al., 2015). Multiple authors found that examples of good practices following the innovation provided by the school enhanced sustainability (e.g., Benz et al., 2004; Andreou et al., 2015). Kirtman (2002) emphasized that support has to be provided through an assessment/evaluation feedback loop that allows for growth, not punishment. This implies that positive reinforcement should be built into the innovation to optimize sustainability.

Characteristics of the individual

Attitudes

Multiple articles (14 in total with varying research designs) reported on the importance of individual stakeholder characteristics such as attitude for sustainable innovations. According to Ajzen (1991) attitude can be defined as one's personal orientation or beliefs related to performing the desired action. Most often mentioned was teacher buy-in and a high level of involvement (e.g., Drits-Esser et al., 2017). Staff having a high level of interest (Gibson and Chase, 2002; King, 2016) and feeling positive toward the innovation and its outcomes is fostering sustainability (e.g., Mouza, 2009; Ng and Nicholas, 2013). On the other hand, resistance to change was found to hinder sustainability (Deaney and Hennessy, 2007). Also, conflicts in personal beliefs (Andreou et al., 2015) and negative feelings toward the innovation (for example the perception that activities created an extra burden for the staff) (Kirtman, 2002; Bambara et al., 2012) hindered sustained implementation of innovations.

Trust

Trust or confidence among staff members was mentioned in eight publications with varying research designs. Trust can be defined as the investment in both one's own and other's reliability, predictability and good intentions (Hargreaves, 2007, p. 187) and has to be present in multiple areas. For example, in their study on ICT practices Deaney and Hennessy (2007) described two internal factors, namely, teachers' technical confidence and their confidence in the innovation. Colleagues' trust levels were mentioned in this perspective as well, with regard to scaling up as an aspect of sustainability. Saito et al. (2012) noted that teachers should have confidence in the effectiveness of the program.

Knowledge and skills

Two studies reported on the influence of knowledge and skills. King (2016) reported on the influence of deep learning on sustainability: the innovation is sustainable if teachers apply procedural and conceptual knowledge to the new practices. These new practices should also meet the needs of their students and should be in alignment with existing practices. In addition, teachers should have obtained the appropriate skills and experience to be able to sustain the innovation (Deaney and Hennessy, 2007).

Context characteristics

Formal external support

Fourteen studies found formal support, including acknowledgment, from outside the organization to influence sustainability. School staff need ongoing training, professional development (Kirtman, 2002; Elias, 2010) and access to higher level support when an innovation does not work for students (Elder and Prochnow, 2016). Over-reliance on local creativity can take a long-term emotional toll on the most committed members (Elias, 2010). The role of training is even bigger in relation to dissemination of practices (Deaney and Hennessy, 2007). Training for staff should be readily available, preferably for longer periods (Ertesvåg and Vaaland, 2007), and the level of support should be adjusted to the different levels of needs, as followers need more support than leaders (Furman Shaharabani and Tal, 2017). Staff also need access to resources, including technical and administrative support (Mouza, 2009). External support in the form of funding also plays an important role in sustainability. When innovations are provided with start-up funds, the stronger innovations tend to survive when this funding is inevitably withdrawn (Owston, 2007). Finally, a strong lead from national policy (Lewin et al., 2009) or plans from schools and school districts (Owston, 2007) influenced sustainability (Payneeandy and Auckloo, 2012). A lack of interest from the state department of education hindered sustainability (Ng and Nicholas, 2013).

Informal external support

Informal external support resulted in higher levels of sustainability, according to 11 studies. Family and student involvement and motivation were found to be important (e.g., Roffe, 2010; Kafyulilo et al., 2016), as was collegial and peer support (Mouza, 2009). The existence of a network of people to support, plan and conduct family and community involvement activities supports student learning and development (Van Voorhis and Sheldon, 2004).

Conclusions and discussion

Core characteristics of sustainability in relation to educational innovations

The literature revealed a large variety of definitions, critical features and working processes for the concept of an educational innovation's sustainability. An innovation refers to ‘The intentional introduction and application within a role (work)group, or organization of ideas, processes, products or procedures new to the relevant unit of adoption [in this paper education], designed to significantly benefit the individual, the group, the organization or wider society’ (West and Farr, 1990, p. 9). Much of this literature have a theoretical focus and influential factors vary between different articles. The aim of this review is to provide an overview of the entire range of factors influencing the sustainability of educational innovations and to enable the development of an overall definition of sustainability based on empirical papers. We performed a systematic literature search of only empirical studies to identify the core elements of sustainability and the factors influencing it, to enable the development of a general model of sustainability. Our search resulted in more than 7,000 papers, of which only 44 publications were left for analysis after applying our inclusion and exclusion criteria to the papers found.

Based on this review, we were able to formulate a definition of sustainability:.‘Sustainability refers to the process of continuing and integrating the innovation's core aspects in organizational routines that are adaptive to ongoing work'

This definition shows that sustainability entails much more than just “continuation” of (the) innovation, not only in the sense that it should become part of organizational routines, but also that adaptivity is important to promote integration into existing practices. Literature shows a shift from fidelity toward a more dynamic interpretation of sustainability focused on adaptivity and continuous adjustment based on the needs of the organization (e.g., see also Fagen and Flay, 2009; Elias, 2010) in order to increase sustainability.

Dissemination of the innovation among members of the organization was often mentioned next to the core aspects of sustainability described above. However, dissemination should not be seen as part of sustainability, but could be considered as being the next phase in the process. Fullan (1992) already described phases of a change process, in which sustainability was regarded as the final phase. Studies in the current review elaborate on the sustainability phase. Owston (2007), for example, distinguished sustainability from transferability in his article on the sustainability of innovative pedagogical practices using technology. He operationally defined sustainability as “the innovation having carried on for a period of more than 2 years without extra fiscal resources; transferability as the innovation having been adopted in its essential form by another grade in that school, school, or school district” (p. 67). Deaney and Hennessy (2007) and Zehetmeier, 2015 described dissemination together with sustainability and made a clear distinction between both concepts. Dissemination should not be confused with knowledge sharing however, which is the process that influences the phases of implementation, sustainability and transferability of the innovation. Dekker and Feijs (2005) distinguish knowledge dissemination and upscaling whereby dissemination is the transfer to other areas and scaling up as the transfer to a larger group of people. It should be noted however, that the change process cannot be regarded as a linear process. Transferability or dissemination of results was regarded in this article as being the next phase of sustainability.

Factors influencing the sustainability of educational innovations

Both hindering and fostering factors influencing sustainability could be categorized into four main categories, in which multiple subcategories could be distinguished. Table 3 summarizes the factors we found in this review to influence the sustainability of innovations.

Much often cited literature refers in general to four main factors that are assumed to influence sustainability of educational innovations. School leadership is crucial for sustainability of educational innovations (e.g., Fullan, 2005; Robinson et al., 2008). Also, collaboration (e.g., Geijsel et al., 2009; Levin and Datnow, 2012), trust (e.g., Sherer and Spillane, 2011; Hargreaves and Fink, 2012) and knowledge sharing (e.g., Gerzon, 2015) and brokerage (e.g., Akkerman and Bruining, 2016) have often been argued to influence sustainability. These factors are mainly situated in the school organizational and the individual domain. However, it also has to be noted here that some of the factors found may slightly overlap and/or are related. For example, Little (1990) conceptualizes collaboration in such way that some of the other factors (e.g., knowledge sharing and brokerage) could also be seen as forms of collaboration.

This review confirms the importance of the factors mentioned above, but also points out additional factors in the context, school organizational and individual domains, as well as an additional category of factors to be considered (i.e., innovation). For example, much evidence has been found in this review for the influence of the organizational members' attitude toward the innovation, but also the structure and perceptions of effectiveness and efficiency of the innovation itself. This review stresses the importance of considering a broader context, instead of focusing on a limited selection of influencing factors. We want to stress here that the general model of the core elements of sustainability and its influencing factors presented in this paper cannot be generalized but should be empirically tested to be able to generalize the findings to other contexts.

Implications for practice

Considering school organizational characteristics, the most evidence was found for the importance of leadership. This is in line with previous research and theoretical articles concerning sustainability in education, in which the role of leadership is emphasized (e.g., Fullan, 2007; Hargreaves and Fink, 2012). However, in this review we were able to identify multiple specific aspects of leadership that have been found to be important.

To influence the sustainability of an innovation, school leaders also need to be knowledgeable about (the core aspect of) the innovation and engage in modeling and networking behavior. Also prioritize the innovation, distribute leadership, communicate a clear vision with regard to the innovation and its place in the school, and facilitate teachers' participation in the innovation, for example, by providing time to participate. Sustaining an educational innovation and its results within schools is often misconceived as something that needs to be done at the end of working with an innovation (Hubers et al., 2019, p. 196). School leaders should focus on sustainability even before the innovation starts, which implies, for example, that they need to think about building a shared vision even before the innovation is implemented. Or, bring more attention to implementation models in schools. The application of implementation science in education is slowly emerging (Albers and Pattuwage, 2017), but here is a gain toward sustainable educational innovation. Moreover, developers should already take into account the aspects found in this review in order to develop sustainable innovations. It should be noted that leaders can be found in multiple organizational layers. “Leaders” could refer to principals, but project leaders, coordinators, or even teacher leaders were described as taking this role in the literature.

Collaboration and knowledge sharing were also identified as influencing sustainability. The focus on collaboration and on knowledge sharing and brokerage within a network was notable. Collaboration in networks implies multiple advantages and actors have access to multiple and diverse types of knowledge and resources (Hubers et al., 2017, 2019). This network-related functioning was not often utilized in the included literature, and may have inhibited sustainability and knowledge dissemination in a broader context.

Innovation characteristics were found to contribute to the sustainability of the innovation. The literature especially pointed to the importance of structured innovations. Hoogland et al. (2016); Hubers et al. (2019) stated in this light that structures and protocols are very important, because they provide a scaffold for teachers to develop their knowledge and skills. Without structure, it is easy for teachers to miss important aspects of the innovation.

Implications considering characteristics of the individual aim to underline the importance of the individuals who carry out the innovations. Attitude, trust, and knowledge and skills are crucial precursors for teachers' active participation in an innovation. For practice, it is important that the members of the organization have a positive attitude toward the innovation and have trust and confidence in colleagues, their leader and their role in the innovation. In order to accomplish this it is important that teachers, for example, are given responsibility and influence at an early stage. Distribution of leadership by the school leader is a crucial aspect in keeping teachers involved with the innovation.

With regard to context characteristics, external (from the team) support from inside, as well as outside the organization were found to be important. It should be noted, however, that this could be very context-dependent and the form of support depends on its relevance for the innovation.

Best evidence synthesis

In order to come to factors with the “biggest influence” on sustainable school improvement, we made a best evidence syntheses to produce and defend conclusions based on the best available evidence, or to conclude that the evidence currently available does not allow for any conclusions (Slavin, 1995). These syntheses are used for reviews based on quantitative datasets. A systematic and transparent framework for assessing confidence in findings of systematic reviews of qualitive research was used (Lewin et al., 2018). This framework considers four components: (1) methodological limitations, (2) coherence, (3) adequacy of data, and (4) relevance. As an example we took only the articles that were affiliated with secondary education As a result, 21 articles met the criteria of methodology, adequacy and relevance. One article was dropped due to a too low score on methodology. Sustainability is influenced by three out of the four main characteristics as elaborated in this review. Leadership is part of the school organizational characteristics, but often came up in the articles and therefor can been taken separately. Factors mentioned most for leadership were knowledgeable leader, support (internal), facilitation an vision.

School organizational factors were collaboration and professional development (support external). Teacher buy-in was a factor most mentioned as an individual characteristic and finally the effectiveness of the intervention within the intervention characteristics. Factors within the context were not mentioned.

Table 4 gives an overview of the factors after best evidence synthesis for secondary education.

Limitations of the study and recommendations for further research

Although this review provides a useful overview of the concept of sustainability of educational innovations and the factors influencing sustainability, we must consider the limitations of this study. First, we were not able to find a lot of empirical studies on the sustainability of innovations. This may have to do with the fact that funding resources often stop after the innovation period, as does the funding for related research. Second, this review covered a variety of different innovations of varied duration, which made it harder to compare and contrast the different innovations and their sustainability. Third, the majority of the studies included were qualitative case report studies. There seems to be a lack of more generalizable large-scale quantitative experimental studies in this field. We found no randomized controlled trials in our studies, and the amount of quasi-experimental studies was limited. Moreover, although this review identified various influential factors, we do not claim that this list of factors is exhaustive. It is possible that there are other factors that influence sustainability that have not yet been studied empirically. In addition to factors found in the review, educational policy by country or continent will be influential as the principles and policy decisions influence the field of education, as well as the collection of laws and rules that govern the operation of educational systems.

There is no hierarchy to be found in the literature, which we would have expected. Different articles on different innovations, come up with different factors of influence. More research is needed on discriminating factors, or how the different factors influence or interact within different forms of education. Further research, for example on empirically testing the model developed in this paper, is also urgently needed. Fourth, although we conducted an extensive literature search, it is possible that we missed some relevant literature. For example, this review has not included unpublished research which may have affected the outcomes. Also, when reviewing the factors and core elements of sustainability we made no distinction between smaller and larger educational innovations. This might have implications for the impact and the kind of influential factors of sustainability.

Furthermore, to ensure that our review only included high quality publications, we focused only on peer-reviewed articles. Therefor we may have missed important information from, for example, books and chapters. Moreover, we ensured the quality of this review by employing detailed, rigorous and explicit methods, focused on two specific research questions (Sackett et al., 2000). Furthermore, we developed and used clear inclusion criteria (Sackett et al., 2000) to overcome possible author biases in selecting literature. We described the methodology used in a detailed manner (Green et al., 2006), and used a scoring system to determine the quality of each publication (Sackett et al., 2000). Because of this rigorous process (Green et al., 2006), we believe that this review can be considered to make a valuable contribution to our knowledge about the sustainability of educational innovations, on which follow-up research into the sustainability of specific innovations can be based.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1

  AjzenI. (1991). The theory of planned behavior. Organ. Behav. Hum. Decis. Process. 50, 179–211. 10.1016/0749-5978(91)90020-T

  • CrossRef
  • Google Scholar

• 2

  AkkermanS.BruiningT. (2016). Multilevel boundary crossing in a professional development school partnership. J. Learn. Sci. 25, 240–284. 10.1080/10508406.2016.1147448

  • CrossRef
  • Google Scholar

• 3

  ^**AlanísI.RodríguezM. A. (2008). Sustaining a dual language immersion program: features of success. J. Latinos Educ. 7, 305–319. 10.1080/15348430802143378

  • CrossRef
  • Google Scholar

• 4

  AlbersB.PattuwageL. (2017). Implementation in Education: Findings from a Scoping Review. Melbourne: Evidence for Learning. Available online at: https://www.newcastle.edu.au/__data/assets/pdf_file/0012/378984/Implementation-in-Education.pdf

  • Google Scholar

• 5

  ^**AndreouT. E.McIntoshK.RossS. W.KahnJ. D. (2015). Critical incidents in sustaining school-wide positive behavioral interventions and supports. J. Spec. Educ. 49, 157–167. 10.1177/0022466914554298

  • CrossRef
  • Google Scholar

• 6

  ^**BambaraL. M.GohA.KernL.CaskieG. (2012). Perceived barriers and enablers to implementing individualized positive behavior interventions and supports in school settings. J. Posit. Behav. Interv. 11, 161–176. 10.1177/1098300712437219

  • CrossRef
  • Google Scholar

• 7

  ^**BeanR. M.DoleJ. A.NelsonK. L.BelcastroE. G.ZigmondN. (2015). The sustainability of a national reading reform initiative in two states. Read. Writ. Q. 31, 30–55. 10.1080/10573569.2013.857947

  • CrossRef
  • Google Scholar

• 8

  ^**BenzM. R.LindstromL.UnruhD.WaintrupM. (2004). Sustaining secondary transition programs in local schools. Remed. Spec. Educ. 25, 39–50. 10.1177/07419325040250010501

  • CrossRef
  • Google Scholar

• 9

  BlaseK.FixsenD. (2013). Core Intervention Components: Identifying and Operationalizing What Makes Programs Work. ASPE Research Brief . US Department of Health and Human Services.

  • Google Scholar

• 10

  BukowitzW. R.WilliamsR. L. (1999). The Knowledge Management Fieldbook. Hoboken, NJ: FT Press.

  • Google Scholar

• 11

  BurkC. (2013). About looking: vision, transformation, and the education of the eye in discourses of school renewal past and present. Br. Educ. Res. J. 36, 65–82. 10.1080/01411920902868413

  • CrossRef
  • Google Scholar

• 12

  CoburnC. E.RusselJ. L.KaufmanJ. H.SteinM. K. (2012). Supporting sustainability: teachers' advice networks and ambitious instructional reform. Am. J. Educ. 119, 137–182. 10.1086/667699

  • CrossRef
  • Google Scholar

• 13

  CoburnC. E.TurnerE. O. (2012). The practice of data use: an introduction. Am. J. Educ. 118, 99–111. 10.1086/663272

  • CrossRef
  • Google Scholar

• 14

  ^**CoffeyJ. H.HornerR. H. (2012). The sustainability of schoolwide positive behavior interventions and supports. Except. Child. 78, 407–422. 10.1177/001440291207800402

  • CrossRef
  • Google Scholar

• 15

  CohenD. K.MehtaJ. D. (2017). Why reform sometimes succeeds: understanding the conditions that produce reforms that last. Am. Educ. Res. J. 54, 1–47. 10.3102/0002831217700078

  • CrossRef
  • Google Scholar

• 16

  CoplandM. (2003). Leadership of inquiry: building and sustaining capacity for school improvement. Educ. Eval. Policy Anal. 25, 375–395. 10.3102/01623737025004375

  • CrossRef
  • Google Scholar

• 17

  DatnowA.ParkV.Kennedy-LewisB. (2012). High school teachers' use of data to inform instruction. J. Educ. Stud. Placed Risk17, 247–265. 10.1080/10824669.2012.718944

  • CrossRef
  • Google Scholar

• 18

  ^**DeaneyR.HennessyS. (2007). Sustainability, evolution and dissemination of information and communication technology-supported classroom practice. Res. Papers Educ. 22, 65–94. 10.1080/02671520601152102

  • CrossRef
  • Google Scholar

• 19

  ^**DekkerT.FeijsE. (2005). Scaling up strategies for change: change in formative assessment practices. Assess. Educ. 12, 237–254. 10.1080/09695940500337215

  • CrossRef
  • Google Scholar

• 20

  ^**Drits-EsserD.Gess-NewsomeJ.StarkL. A. (2017). Examining the sustainability of teacher learning following a year-long science professional development programme for inservice primary school teachers. Prof. Dev. Educ. 43, 375–396. 10.1080/19415257.2016.1179664

  • CrossRef
  • Google Scholar

• 21

  ^**Edwards GrovesC.RönnermanK. (2013). Generating leading practices through professional learning. Prof. Dev. Educ. 39, 122–140. 10.1080/19415257.2012.724439

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 22

  ^**ElderK. I.ProchnowJ. E. (2016). PB4L school-wide: what will support the sustainability of the initiative?N. Z. J. Educ. Stud. 51, 83–97. 10.1007/s40841-016-0036-1

  • CrossRef
  • Google Scholar

• 23

  ^**EliasM. (2010). Sustainability of social-emotional learning and related programs: lessons from a field study. Int. J. Emot. Educ. 2, 17–33. Available online at: https://www.um.edu.mt/library/oar//handle/123456789/6099

  • Google Scholar

• 24

  ^**ErtesvågS. K.RolandP.VaalandG. S.StørksenS.VelandJ. (2010). The challenge of continuation: schools' continuation of the Respect program. J. Educ. Change. 11, 323–344. 10.1007/s10833-009-9118-x

  • CrossRef
  • Google Scholar

• 25

  ^**ErtesvågS. K.VaalandG. S. (2007). Prevention and reduction of behavioural problems in school: an evaluation of the Respect program. Educ. Psychol. 27, 713–736. 10.1080/01443410701309258

  • CrossRef
  • Google Scholar

• 26

  FagenM. C.FlayB. R. (2009). Sustaining a school-based prevention program: results from the Aban Aya sustainability project. Health Educ. Behav. 36, 9–23. 10.1177/1090198106291376

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27

  FeldmanM. S.PentlandB. T. (2003). Reconceptualizing organizational routines as a source of flexibility and change. Adminstr. Sci. Q. 48, 94–118. 10.2307/3556620

  • CrossRef
  • Google Scholar

• 28

  ^**FergusonN.CurrieL.-A.PaulM.ToppingK. (2011). The longitudinal impact of a comprehensive literacy intervention. Educ. Res.53, 237–256. 10.1080/00131881.2011.598657

  • CrossRef
  • Google Scholar

• 29

  FullanM. (1992). Successful School Improvement. Michigan: McGraw-Hill Education.

  • Google Scholar

• 30

  FullanM. (2005). Leadership and Sustainability. Thousand Oaks, CA: Corwin Press.

  • Google Scholar

• 31

  FullanM. (2006). Turn Around Leadership. San Francisco, CA; Totonto: ON: Jossey-Bass.

  • Google Scholar

• 32

  FullanM. (2007). The New Meaning of Educational Change, 4th Edn. London: Taylor and Francis Ltd.

  • Google Scholar

• 33

  ^**Furman ShaharabaniY.TalT. (2017). Teachers' practice a decade after an extensive professional development program in science education. Res. Sci. Educ. 47, 1031–1053. 10.1007/s11165-016-9539-5

  • CrossRef
  • Google Scholar

• 34

  ^**GaikhorstL.BeishuizenJ. J. J.ZijlstraB. J. H.VolmanM. L. L. (2017). The sustainability of a teacher professional development programme for beginning urban teachers. Cambridge J. Educ. 47, 135–154. 10.1080/0305764X.2015.1125449

  • CrossRef
  • Google Scholar

• 35

  GeijselF. P.SleegersP. J.StoelR. D.Kr?gerM. L. (2009). The effect of teacher psychological and school organizational and leadership factors on teachers' professional learning in Dutch schools. Element. School J.109, 406–427. 10.1086/593940

  • CrossRef
  • Google Scholar

• 36

  GerzonN. (2015). Structuring professional learning to develop a culture of data use: aligning knowledge from the field and research findings. Teach. Coll. Rec. 117, 1–28. 10.1177/016146811511700407

  • CrossRef
  • Google Scholar

• 37

  ^**GibsonH. L.ChaseC. (2002). Longitudinal impact of an inquiry-based science program on middle school students' attitudes toward science. Sci. Educ. 86, 693–705. 10.1002/sce.10039

  • CrossRef
  • Google Scholar

• 38

  ^**Gilad-HaiS.SomechA. (2016). The day after: the organizational consequences of innovation implementation in experimental schools. J. Educ. Administr. 54, 19–40. 10.1108/JEA-07-2014-0084

  • CrossRef
  • Google Scholar

• 39

  GouletL.KrentzC.ChristiansenH. (2003). Collaboration in education: the phenomenon and process of working together. Alberta J. Educ. Res.49, 325–40. 10.11575/ajer.v49i4.55027

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40

  GreenB. N.JohnsonC.AdamsA. (2006). Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J. Chiropractic Med. 5, 101–117. 10.1016/S0899-3467(07)60142-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 41

  HargreavesA. (2007). “Sustainable professional learning communities,” in Professional Learning Communities: Divergence, Depth and Dilemmas. p. 181–195.

  • Google Scholar

• 42

  HargreavesA.FinkD. (2008). Distributed leadership: democracy or delivery?J. Educ. Admin.46, 229–240. 10.1108/09578230810863280

  • CrossRef
  • Google Scholar

• 43

  HargreavesA.FinkD. (2012). Sustainable Leadership.6th Edn. London: John Wiley and Sons.

  • Google Scholar

• 44

  HarrisA.DeFlaminisJ. (2016). Distributed leadership in practice: evidence, misconceptions and possibilities. Manag. Educ. 30, 141–146. 10.1177/0892020616656734

  • CrossRef
  • Google Scholar

• 45

  HarrisA.SpillaneJ. (2008). Distributed leadership through the looking glass. Manag. Educ. 22, 31–34. 10.1177/0892020607085623

  • CrossRef
  • Google Scholar

• 46

  HooglandI.SchildkampK.Van der KleijF. M.HeitinkM. C.Kippers W. B.VeldkampB. P.et al. (2016). Prerequisites for data-based decision making in the classroom: Research evidence and practical illustrations. Teach. Teach. Educ. 60, 377–386. 10.1016/j.tate.2016.07.012

  • CrossRef
  • Google Scholar

• 47

  HubersM. D.MoolenaarN. M.SchildkampK.DalyA. J.HandelzaltsA.PietersJ. M. (2017). Share and succeed: the development of knowledge sharing and brokerage in data teams' network structures. Res. Paper Educ. 33, 1–23. 10.1080/02671522.2017.1286682

  • CrossRef
  • Google Scholar

• 48

  HubersM. D.PoortmanC. L.SchildkampK.PietersJ. M. (2019). Spreading the word: Boundary crossers building collective capacity for data use. Teach. Coll. Record. 121, 1–45. 10.1177/016146811912100102

  • CrossRef
  • Google Scholar

• 49

  ^**JessonR.LimbrickL. (2014). Can gains from early literacy interventions be sustained? The case of Reading Recovery. J. Res. Read. 37, 102–117. 10.1111/1467-9817.12017

  • CrossRef
  • Google Scholar

• 50

  ^**KafyuliloA.FisserP.VoogtJ. (2016). Factors affecting teachers' continuation of technology use in teaching. Educ. Inform. Technol. 21, 1535–1554. 10.1007/s10639-015-9398-0

  • CrossRef
  • Google Scholar

• 51

  ^**KingF. (2016). Teacher professional development to support teacher professional learning: systemic factors from Irish case studies. Teach. Dev. 20, 574–594. 10.1080/13664530.2016.1161661

  • CrossRef
  • Google Scholar

• 52

  ^**KirtmanL. (2002). Policy and practice: restructuring teachers' work. Educ. Policy Anal. Arch. 10:25. 10.14507/epaa.v10n25.2002

  • CrossRef
  • Google Scholar

• 53

  ^**LarsenT.SamdalO. (2008). Facilitating the implementation and sustainability of Second Step. Scand. J. Educ. Res.52, 187–204. 10.1080/00313830801915820

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 54

  LawsonH. A. (2004). The logic of collaboration in education and the human services. J. Interprof. Care18, 225–237. 10.1080/13561820410001731278

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 55

  LevinJ. A.DatnowA. (2012). The principal role in data-driven decision making: using case-study data to develop multi-mediator models of educational reform. Sch. Eff. Sch. Improv. 23, 179–201. 10.1080/09243453.2011.599394

  • CrossRef
  • Google Scholar

• 56

  ^**LewinC.ScrimshawP.SomekhB.HaldaneM. (2009). The impact of formal and informal professional development opportunities on primary teachers' adoption of interactive whiteboards. Technol. Pedag. Educ. 18, 173–185. 10.1080/14759390902992592

  • CrossRef
  • Google Scholar

• 57

  LewinS.BoothA.GlentonC.Munthe-KaasH.RashidianA.WainwrightM.et al. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement. Sci.13, 1–10. 10.1186/s13012-017-0688-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58

  LittleJ. W. (1990). The persistence of privacy: autonomy and initiative in teachers' professional relations. Teach. Coll. Rec. 91, 509–536. 10.1177/016146819009100403

  • CrossRef
  • Google Scholar

• 59

  LittleJ. W. (2010). Social Network Theory and Educational Change. Harvard Education Press.

  • Google Scholar

• 60

  MartinM.WilkinsonJ. E.McPheeA.McQueenI.McConnellF.BaronS. (2006). Implementing critical skills in UK schools. J. Educ Teach. 32, 423–434. 10.1080/02607470600982076

  • CrossRef
  • Google Scholar

• 61

  ^**MathewsS.McIntoshK.FrankJ. L.MayS. L. (2014). Critical features predicting sustained implementation of school-wide positive behavioral interventions and supports. J. Posit. Behav. Interv. 16, 168–178. 10.1177/1098300713484065

  • CrossRef
  • Google Scholar

• 62

  MoolenaarN. M.SleegersP. J. C.DalyA. J. (2012). Teaming up: linking collaboration networks, collective efficacy, and student achievement. Teach. Teach. Educ. 28, 251–262. 10.1016/j.tate.2011.10.001

  • CrossRef
  • Google Scholar

• 63

  ^**MouzaC. (2009). Does research-based professional development make a difference? A longitudinal investigation of teacher learning in technology integration. Teach. Coll. Rec. 111, 1195–1241. 10.1177/016146810911100502

  • CrossRef
  • Google Scholar

• 64

  ^**NgW.NicholasH. (2013). A framework for sustainable mobile learning in schools. Br. J. Educ. Technol.44, 695–715. 10.1111/j.1467-8535.2012.01359.x

  • CrossRef
  • Google Scholar

• 65

  ^**OwstonR. (2007). Contextual factors that sustain innovative pedagogical practice using technology: an international study. J. Educ. Change8, 61–77. 10.1007/s10833-006-9006-6

  • CrossRef
  • Google Scholar

• 66

  ^**PayneeandyS.AucklooP. (2012). School-based teacher training and the development of literacy in low-achieving schools. Int. J. Learn. 18, 269–282. 10.18848/1447-9494/CGP/v18i10/47779

  • CrossRef
  • Google Scholar

• 67

  PetersJ. (2011). Sustaining school colleagues' commitment to a long-term professional experience partnership. Australian J. Teacher Educ. 36, 16–30. 10.14221/ajte.2011v36n5.2

  • CrossRef
  • Google Scholar

• 68

  PetticrewM.RobertsH. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA; Oxford: Blackwell Publishers.

  • Pubmed Abstract
  • Google Scholar

• 69

  ^**PinkelmanS. E.McintoshK.RasplicaC. K.BergT.Strickland-CohenM. K. (2015). Perceived enablers and barriers related to sustainability of school-wide positive behavioral interventions and supports. Behav. Disord. 40, 171–183. 10.17988/0198-7429-40.3.171

  • CrossRef
  • Google Scholar

• 70

  ^**PostholmM. B. (2011). A completed research and development work project in school: the teachers' learning and possibilities, premises and challenges for further development. Teach. Teach. Educ. 27, 560–568. 10.1016/j.tate.2010.10.010

  • CrossRef
  • Google Scholar

• 71

  RikkerinkM. (2011). Invoering van een gedigitaliseerde onderwijspraktijk - Deel A. Patronen van interventies in een model van organisatieleren en leiderschapspraktijken [Implementation of a digital teaching practice—Part A. Patterns of interventions in a framework of organizational learning and leadership practices] [Dissertation], Utrecht University, Utrecht, Netherlands.

  • Google Scholar

• 72

  RobinsonV. M. J.LloydC. A.RoweK. J. (2008). The impact of leadership on student outcomes: an analysis of the differential effects of leadership types. Educ. Administr. Q. 44, 635–674. 10.1177/0013161X08321509

  • CrossRef
  • Google Scholar

• 73

  ^**RoffeI. (2010). Sustainability of curriculum development for enterprise education: observations on cases from Wales. Educ. Train. 52, 140–164. 10.1108/00400911011027734

  • CrossRef
  • Google Scholar

• 74

  ^**SackettD. L.StrausS. E.RichardsonW. S.RosenbergW.HaynesR. B. (2000). Evidence-Based Medicine: How to Practice and Teach EBM.2nd Edn. New York, NY: Churchill Livingstone.

  • Pubmed Abstract
  • Google Scholar

• 75

  ^**SaitoE.KhongT. D. H.TsukuiA. (2012). Why is school reform sustained even after a project? A case study of Bac Giang Province, Vietnam. J. Educ. Change13, 259–287. 10.1007/s10833-011-9173-y

  • CrossRef
  • Google Scholar

• 76

  ^**SandersM. G. (2009). Collaborating for change: how an urban school district and a community-based organization support and sustain school, family, and community partnerships. Teach. Coll. Rec. 111, 1693–1712. 10.1177/016146810911100703

  • CrossRef
  • Google Scholar

• 77

  ^**SandersM. G. (2012). Sustaining programs of school, family, and community partnerships. Educ. Policy26, 845–869. 10.1177/0895904811417591

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 78

  ^**SandholtzJ. H.RingstaffC. (2016). The influence of contextual factors on the sustainability of professional development outcomes. J. Sci. Teacher Educ. 27, 205–226. 10.1007/s10972-016-9451-x

  • CrossRef
  • Google Scholar

• 79

  ShererJ.SpillaneJ. P. (2011). Constancy and change in work practice in schools: the role of organizational routines. Teach. Coll. Rec. 113:611–57. 10.1177/016146811111300302

  • CrossRef
  • Google Scholar

• 80

  SlavinR. E. (1995). Best evidence synthesis: an intelligent alternative to meta-analysis. J. Clin. Epidemiol. 48, 9–18. 10.1016/0895-4356(94)00097-A

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 81

  SpillaneJ. P. (2006). “Towards a theory of leadership practice: A distributed perspective,” in Rethinking Schooling (Routledge), 208–242.

  • Google Scholar

• 82

  SpillaneJ. P. (2009). Managing to lead: reframing school leadership and management. Phi Delta Kappan91, 70–73. 10.1177/003172170909100315

  • CrossRef
  • Google Scholar

• 83

  SpillaneJ. P. (2012). Data in practice: conceptualizing the data-based decision-making phenomena. Am. J. Educ. 118, 113–141. 10.1086/663283

  • CrossRef
  • Google Scholar

• 84

  SpillaneJ. P.CamburnE. M.PustejovskyJ.ParejaA. S.LewisG. (2008). Taking a distributed perspective. Epistemological and methodological tradeoffs in operationalizing the leader-plus aspect. J. Educ. Administr. 46, 189–213. 10.1108/09578230810863262

  • CrossRef
  • Google Scholar

• 85

  SpillaneJ. P.DiamondJ. B. (2007). Distributed Leadership in Practice. New York, NY: Teachers College; Columbia University.

  • Google Scholar

• 86

  SpillaneJ. P.HalversonR.DiamondJ. B. (2004). Towards a theory of leadership practice: a distributed perspective. J. Curr. Stud. 36, 3–34. 10.1080/0022027032000106726

  • CrossRef
  • Google Scholar

• 87

  StolpS.SmithS. C. (1994). School Culture and Climate: The Role of the Leader. OSSC Bulletin. Eugene: Oregon School Study Council.

  • Google Scholar

• 88

  ^**StringfieldS.ReynoldsD.SchafferE. C. (2008). Improving secondary students' academic achievement through a focus on reform reliability: 4- and 9-year findings from the High Reliability Schools project. Sch. Eff. Sch. Improv. 19, 409–428. 10.1080/09243450802535190

  • CrossRef
  • Google Scholar

• 89

  ^**TamF. W. (2009). Sufficient conditions for sustainable instructional changes in the classroom: the case of Hong Kong. J. Educ. Change10, 315–336. 10.1007/s10833-008-9091-9

  • CrossRef
  • Google Scholar

• 90

  TappelA. P. M.PoortmanC. L.SchildkampK.VisscherA. J. (In Press). Distinguisihing aspects of sustainability. J. Educ. Chang.

  • Google Scholar

• 91

  ThoonenE. E. J.SleegersP. J. C.OortF. J.PeetsmaT. T. D.GeijselF. P. (2011). How to improve teaching practices: the role of teacher motivation, organizational factors and leadership practices. Educ. Administr. Q. 47, 496–536. 10.1177/0013161X11400185

  • CrossRef
  • Google Scholar

• 92

  ^**Van VoorhisF.SheldonS. (2004). Principals' roles in the development of US programs of school, family, and community partnerships. Int. J. Educ. Res. 41, 55–70. 10.1016/j.ijer.2005.04.005

  • CrossRef
  • Google Scholar

• 93

  WestM. A. (2012). Effective Teamwork; Practical Lessons from Organizational Research. New York, NY: John Wiley and Sons and The British Psychology Centre.

  • Google Scholar

• 94

  WestM. A.FarrJ. L. (1990). “Innovation at work,” in Innovation and Creativity at Work, eds M. A. West and J. L. Farr (New York, NY: John Wiley and Sons), 3–13.

  • Google Scholar

• 95

  Wiltsey StirmanS.KimberlyJ.CookN. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement. Sci. 7, 1–19. 10.1186/1748-5908-7-17

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 96

  ^**WoodsP. A.RobertsA. (2016). Distributed leadership and social justice: images and meanings from across the school landscape. Int. J. Leaders. Educ. 19, 138–156. 10.1080/13603124.2015.1034185

  • CrossRef
  • Google Scholar

• 97

  ^**YoungsP.KingM. B. (2002). Principal leadership for professional development to build school capacity. Educ. Administr. Q. 38, 643–670. 10.1177/0013161X02239642

  • CrossRef
  • Google Scholar

• 98

  ^**ZehetmeierS. (2015). Sustaining and scaling up the impact of professional development programmes. ZDM47, 117–128. 10.1007/s11858-015-0671-x

  • CrossRef
  • Google Scholar