Developing a Self-Reflection Tool to Assess Schools’ Openness

Sotiriou, Menelaos; Sotiriou, Sofoklis; Bogner, Franz X.

doi:10.3389/feduc.2021.714227

ORIGINAL RESEARCH article

Front. Educ., 04 November 2021

Sec. Assessment, Testing and Applied Measurement

Volume 6 - 2021 | https://doi.org/10.3389/feduc.2021.714227

Developing a Self-Reflection Tool to Assess Schools’ Openness

Menelaos Sotiriou¹*

Sofoklis Sotiriou²

Franz X. Bogner³

¹Research and Development Department, Science View, Athens, Greece
²Research and Development Department, Ellinogermaniki Agogi, Pallini – Attica, Greece
³Centre of Math and Science Education, University of Bayreuth, Bayreuth, Germany

In 2015, the EU Educational Policy document “Science Education for Responsible Citizenship” introduced the concept of Open Schooling as a promising approach to transform schools into innovation hubs within their local communities. In an open school environment, external ideas need to challenge traditional internal views and, in turn, to benefit its students as the community it serves. Such engaging environments may vitally contribute to their community when students’ projects introduce real needs into a community outside of school, present them publicly, and enrich local expertise. Additionally, such school environments may foster learner independence—and interdependence—through collaboration and through the provision of opportunities for learners to understand and interrogate their place in the world. In our study, it was realised in the framework of the European Union–supported Coordination Action called “Open Schools for Open Societies” (OSOS), and we have analysed the development process of a large number of schools with a Self-Reflection Tool (OSOS-SRT), focusing on the organisational change during the implementation of the Open Schooling approach. Rooted in the theory of RRI (Responsible Research and Innovation) principles, the tool has shown its potential to analyse the openness level of each school. In this study, we are presenting the validation of the proposed instrument and empirical data from the overall transformation process. The school environments were evaluated for one full academic year by completing two measurements, one before their involvement and one after their engagement in the transformation journey. Participating schools achieved an average increase of 11.34% in their openness levels, while based on the first measurement the lowest performers achieved a 35–45% increase while the higher scorer still reached a small but significant increase in the recorded openness levels. Our findings could act as a reference point for educational policy actions to support school development through cooperation and continuous interaction with external stakeholders. Consequences for school management, development plans, and teachers’ professional development are discussed.

Introduction

Numerous education reform initiatives in Europe try to make schools more effective and provide an education that prepares students for life in the 21st century. Schools are being asked to increase the quality of their services, notably by providing more students with advanced skills and the ability to be flexible thinkers and problem solvers (OECD, 2020a). Current governmental initiatives apply large-scale ambitious plans to rebuild and remodel schools in creating open and creative learning environments. They intend to support young people to unlock hidden talents and reach their full potential, to provide teachers with 21st-century workplaces, and to provide access to facilities useable by all members of the local community (EC, 2015a; EC, 2015b; EC, 2016a; EC, 2016b; EC, 2019a). All these efforts clearly serve—at a different level—the vision of re-schooling, towards schools as “Core Social Centres” and “Focused Learning Organisations,” strong, dynamic establishments in strong cultures of equity and consensus about their value, following system-wide, root-and-branch reform as it was earlier proposed back by the International Schooling for Tomorrow Forum (OECD, 2006). In the OECD re-schooling scenarios, schools are revitalised around a strong “knowledge” agenda, with far-reaching implications for the organization of individual institutions and for the system as a whole. The academic/artistic/competence development goals are paramount; experimentation and innovation are the norms. Curriculum specialists flourish as do innovative forms of assessment and skills recognition. All this takes place in an environment where quality norms rather than accountability measures are the primary means (Biesta, 2009; Bottery, 2012). Professionals (teachers and other experts) would in general be highly motivated, and they work in environments characterized by the continuing professional development of personnel, group activities, and networking. In these environments, a strong emphasis is placed on educational research and development (EC 2017a; EC 2017b; EC 2017c).

The OECD re-schooling scenarios (OECD, 2006) describe a substantial strengthening of schools with new dynamism, recognition, and purpose by formulating guiding principles for an Open Schooling environment that is supposed to promote deeper learning (Fullan and Langworthy, 2014; Fullan and Scott, 2014; Tichnor-Wagner et al., 2017). The current OECD re-schooling scenarios (OECD 2020a) claim the open school culture as a strong driver of developing the future paths for schooling. Using these scenarios may help to identify opportunities and challenges for future schooling and better to prepare and act now: “Opening the ‘school walls’ connects schools to their communities, favouring ever-changing forms of learning, civic engagement, and social innovation” (p.49, OECD, 2020a).

Becoming an Open School cannot be seen as an isolated “project”—it demands a root-and-branch rethink, not just in pedagogy, but in every aspect of the way, the school is organised: its structure, culture, and the use of space, place, and time. An Open School as an open, curious, welcoming, and democratic environment is supposed to support the development of innovative and creative educational activities (Feddar et al., 2017). A successful environment needs facilitating the process for envisioning and managing change in school settings by providing a simple and flexible structure to follow (Paul Hamlyn Foundation, 2012; Hamilton, 2015). Innovative ways do not simply automate processes, they need to inspire, engage, connect, and provide a framework for school leaders to engage, discuss, and explore (Martinez and McGrath, 2014). They need to offer answers on how schools can evolve, transform, and reinvent; how schools facilitate open, more effective and efficient co-design, co-creation, and use of educational content (both from formal and informal providers), tools and services for personalized learning and teaching; how schools can become innovation incubators and accelerators (Winthrop et al., 2017).

Aims of the Study

The first aim of our work was to prove the effectiveness and efficiency of the proposed instrument for its designed role (see chapter 2). The second aim was to pilot the use of the instrument with a large number of schools from different countries across Europe. These schools had to follow the open schooling approach for at least one academic year. During this period, the schools had to adopt the Open School Roadmap (Sotiriou et al., 2017a) that describes the overall approach towards openness and offers guidance for school heads and teachers to introduce an open school culture to their settings. It is a step-by-step guide that presents the characteristics of an open school environment and the key features of the educational activities that are taking place. It also describes the necessary actions at the management level, the processes that must be in place, and the competencies of the teachers to support the operation of the school. Our study, independently of the exact level of school autonomy, assumes that the school heads have the responsibility for that school development as far as the integration of innovative projects and approaches in the school curriculum. The proposed activities that were implemented in the participating schools were designed and adopted to take into consideration the local situation and the overall framework in each country. In all cases, school heads had the flexibility to allocate the required time in the proposed open schooling activities. In the participating countries, there is in place a standardised policy for science-related subjects taught at school (including a school curriculum with shared instructional materials, and staff development and training) that tends to be mandatory and regulated at the district or national level, while schools encourage and make available teacher mentoring on their own initiative (OECD, 2020b).

The Process of Assessing Schools’ Openness

Characteristics of the Open Schools

Open schools set forward an innovation agenda that has the following characteristics (Sotiriou et al., 2017a):

• Promotes the collaboration with non-formal and informal education providers, enterprises, and civil society enhanced to ensure relevant and meaningful engagement of all societal actors with science and increase the uptake of science studies and science-based careers, employability, and competitiveness (Goddard at al., 2015; Sotiriou et al., 2017b; Wenner and Campbell, 2017). Individual schools are working with science centres and museums, industries, research institutes, universities, national school networks, and teacher training associations in an innovative collaboration scheme towards the introduction of open schooling approaches through a bottom-up approach. With the focus on science learning in both primary and secondary education levels, the project proposes new and diverse models of collaborations between the above-mentioned stakeholders (Tichnor-Wagner et al., 2017). By building on the best of current practice, such an approach aims to take us beyond the constraints of present structures of schooling toward a shared vision of excellence (Doten-Snitker et al., 2020).

• Supports schools to become an agent of community well-being. By creating a model of collaboration with local stakeholders and by using activities that require the involvement of different actors, open schools are linked with their local communities in a much deeper level. Students’ projects are focusing on local needs and challenges, including studies and analysis of the opportunities offered and proposing solutions based on analysis of scientific data (Winthrop et al., 2017).

• Promotes partnerships that foster expertise, networking, sharing, and applying science and technology research findings and that bring real-life projects to the classroom. Open schools are supposed to develop and promote innovative educational applications, share and apply frontier research findings, support competencies through creative problem-solving, discovery learning by doing, experiential learning, critical thinking, and creativity, including projects and activities that simulate the real scientific work (Stilgoe et al., 2020). Examples can be seen in nanotechnology applications in different sectors, organic farming, and healthy food, implementing a project with aerospace industry, analyse data from large research infrastructures like CERN or robotic telescopes’ networks (Kourkoumelis and Vourakis, 2016; Wietsma et al., 2018). In the open school environment, the afore-mentioned scenarios are becoming part of curriculum-led learning (integrating/embedding them in the everyday school practice) while extra-curricular activities (e.g., visits to museums, science centres, research centres, field trips) are coupled with home- and community-centred (informal) learning experiences. Each open school brings together representatives from industry and civil society associations who—in cooperation with school community—scan the horizons, analyse the school and community needs, and cooperate to design common projects and to propose innovative solutions (MacBeath and Mortimore, 2001).

• Focuses on Effective Parental Engagement. The Open Schooling approach is suggesting four courses of action: a) effective parental engagement needs planning for and embedded in a whole school or service strategy (Rogers et al., 2009). The planning cycle will include a comprehensive needs’ analysis, the establishment of mutual priorities, ongoing evaluation of interventions, and a public awareness process to help parents and teachers understand and commit to the Open School Development plan. b) Effective leadership of parental engagement is essential to the success of the Open Schooling strategies. A parental engagement programme is often led by a senior leader although leadership may also be distributed in the context of a programme or cluster of schools and services working to a clear strategic direction. c) Collaboration and engagement: Parental engagement requires active collaboration with parents and should be proactive rather than reactive. It should be sensitive to the circumstances of all families, recognises the contributions parents can make, and aims to empower parents. d) Sustained improvement: A parental engagement strategy should be the subject of ongoing support including strategic planning which embeds parental engagement in whole-school development plans, sustained support, resourcing and training, community involvement at all levels of management, and a continuous system of evidence-based development and review.

• Teaching science for difference: Gender Issues. The Open Schooling approach recommends replacing the competitive-type classroom environment by more and more girls’ friendly instructional approach in which enough time and conditions are given to think, inquire, and understand thoroughly. Instructional methods that foster students’ understanding while decreasing competitiveness in science classes might contribute to girls’ participation and performance in advanced science classes while also supporting the learning of many boys (OECD, 2019). This could be accomplished by, for example, sharing ideas, arguing, asking questions, and analysing data in small groups of students who work in a collaborative manner. This is an approach that clearly reduces the competitive nature of the whole classroom (teacher-centred) approach. The Open Schooling educational activities and projects are based on pedagogical approaches that produce the outcome of proportional participation of both genders.

Design Features of the Open Schooling Activities

The Open Schooling approach builds on the essential features of creative learning including exploration, dynamics of discovery, student-led activity, engagement in scientifically oriented questions, priority to evidence in responding to questions, formulations of evidence-based explanations, the connection of explanations to scientific knowledge, and communication and justification of explanations (Sotiriou et al., 2017b; Sotiriou et al., 2020). These elements support creativity as a generic element in the processual and communicative aspects of the pedagogy and proposing innovative teaching strategies that will offer students high participation and enable them to generate highly imaginative possibilities (Covay and Carbonaro, 2010). At the same time, the Open Schooling approach is based on the main principles of the Responsible Research and Innovation process: learners’ engagement, unlock of their full potential, sharing results, and provide access to scientific archives, designing innovative activities for all. Based on that, the Open Schools promote a series of educational activities in the form of real-life projects that utilize innovative ideas and creativity and empower students to actively engage themselves in the learning process and improve their conceptual understanding of various scientific topics (Sotiriou et al., 2020). The traditional school environment (along with the curriculum constraints and the time limitations set by strict timetables) is acting as an obstacle to the introduction of innovation in the school setting, and it sets numerous limitations to the creative involvement of students. At the same time, the engagement of external stakeholders is restricted. In an Open schooling, the environment is therefore intended that the presented educational practices and strategies may allow science educators and specifically late primary and early secondary school teachers to identify creative activities for teaching science, considering the active involvement of scientists, experts, industrial partners, and community stakeholders who feel that they share responsibility for students learning. Furthermore, the Open Schooling approach aims to enable teachers to either create new creative activities or to properly assemble parts of different educational activities into interdisciplinary learning scenarios. The Open Schooling activities have the following four characteristics (Sotiriou et al., 2017a). They must be:

• Placed: The activity is located, either physically or virtually, in a world that the student recognizes and is seeking to understand.

• Purposeful: The activity feels authentic, and it absorbs the student in actions of practical and intellectual value and fosters a sense of agency.

• Passion-led: The activity enlists the outside passions of both students and teachers, enhancing engagement by encouraging students to choose areas of interest that matter to them.

• Pervasive: The activity enables the student to continue learning outside the classroom, drawing on family members, peers, local experts, and online references as sources of research and critique.

These four criteria can provide a useful checklist for teachers formulating their learning designs but also suggest what a science classroom and a school as an organization needs to offer to become more engaging in itself: a place-based curriculum, purposeful projects, passion-led teaching and learning, and pervasive opportunities for research and constructive challenge.

Methodology of the Study

Method of Assessing the Effectiveness of the Open Schooling Approach

Assessing the effectiveness of the Open Schooling approach to schools needs an appropriate tool that is sensitive to analyse the key characteristics of these environments (Fullan and Quinn 2016; Fullan, 2018). The development of the Open Schools Self-Reflection Tool (OSOS-SRT) was the response to follow the organisational change during the implementation. Based on the recent methodologies for assessing the impact of RRI in education (Hobbiss et al., 2019; van Atteveldt et al., 2019), it analysed the school community engagement in research and innovation. Furthermore, it is supposed to enable easier access to scientific results for students, to take up gender and ethics in the research and innovation content and process, and to act as a bridge of formal and informal science education, on the EU recommendations for the science education and on the development of responsible citizenship (EC, 2015a) as well as to analyse schools’ e-maturity (Kampylis et al., 2015; Sotiriou et al., 2016; EC, 2019b). By focusing on three identified areas of “growth—school management, school process, and teachers’ professional development—the specific instrument is offering the opportunity to the school community stakeholders to describe in detail the current situation in their school while at the same time, they are able to translate the findings to specific recommendation for future actions and development. More specifically, the tool aims to support the school heads to identify the status of their school in the following key areas (levels of openness):

• Management: The aim of the instrument is to assess the vision, the leadership of the school community key stakeholders towards the adoption of a changing culture towards openness (Earley and Greany, 2017; Hobbiss et al., 2019), and the overall innovation potential of the school community (George and Desmidt, 2018) and to highlight the appearance (or not) of the key factors that can catalyse the cultural changes (e.g., coherence of local or national policies, development of a shared vision and understanding, development of motivation mechanisms and specific plans for staff competencies, school autonomy). This section of the instrument includes a step-by-step approach for the school heads to define a root of development and to locate the current position of their school in the innovation journey.

• Process: The aim of this section is to identify which process is already in place in the school community and which must be further developed. The instrument is sensitive in highlighting the processes and the mechanisms such as a) the operation of collaborative environments and tools (for content co-creation and sharing), b) how many members of the school community are using them regularly, c) adjustments with the curriculum that allow for the implementation of open school activities (as defined in the previous section), d) parents and external stakeholders’ involvement in the open school activities, and e) procedures in place that are offering opportunities to reflect and debate, communication and feedback mechanisms.

• Professional Development: The aim of this section of the instrument is to assess to what extent teachers and school staff engaged in the open schooling approach have a holistic view of science, scientific research, and major scientific developments (Harris and Tassell, 2005; Sotiriou et al., 2016). This section includes reflections on the integration of RRI principles into school curricula and teaching practices (van Atteveldt et al., 2019). These reflections and evaluation of curricula and practices are supposed to reveal changes in awareness/knowledge aspects/behaviour in relation to the RRI principles—such as gender, ethics, open access, open science, public engagement, governance, socio-economic development and sustainability, social issues related to scientific developments. Supporting teacher leadership may play an essential role to empower reaching this target (Muijs and Harris, 2003).

For each one of the above-mentioned levels, the tool reflects upon eight task-specific statements (24 statements in total, see Table 1) that follow specific Open Schooling indicators (Sotiriou et al., 2017a—see Supplementary Annex S1).

TABLE 1

TABLE 1. The eight items in each one of the three levels of openness, 24 task-specific statements in total.

In our methodology and while we were developing the Self-Reflection Tool, we choose to use a form of a task-specific rubric (see Supplementary Annex S2). Task-specific statements function as “scoring directions” for the person who is grading the work. Because they detail the elements to look for in a participant’s answer to a particular task, scoring participants’ responses (in our case schools) with task-specific statements is lower-inference work than scoring participants’ responses with general rubrics. For this reason, it is faster to train raters to reach acceptable levels of scoring reliability using task-specific rubrics for large-scale assessment. Similarly, it is easier for school heads to apply task-specific rubrics consistently with a minimum of practice (Brookhart 2013). Rubrics have been tested for their validity as an assessment tool (self-assessment as well) for peer groups of participants (Hafner and Hafner, 2003).

In Supplementary Annex S2, the 24 task-specific statements are presented in detail.

For each statement in each level, the school head or the school representative can choose one statement that corresponds to the actual situation of the school at a specific time. In our study, the same person (school head) answered both the pre- and post-questionnaire. Each statement corresponds to a school typology, according to the school’s readiness to adopt an open schooling culture. The four school typologies that are presented in Table 2 are following the Open School characteristics that were presented in the previous section. Schools, according to their statements, will be characterized according to their openness status to four categories: Enabled, Consistent, Integrated, or Advanced.

TABLE 2

TABLE 2. The four school typologies according to the Open School characteristics.

After the completion of each one of the required sections of the self-reflection tool, the school representative gets a report that includes the answers in each one of the sections as well as the results of the reflection process. The status of the school could be Enabled (25% as the minimum of the selected scales—starting point for a school unit), Consistent (scores between 26 and 50%), Integrated (scores 51–75%), or Advanced (scores 76–100%).

The development of the statements was realised in two phases. First were developed according to the above-mentioned steps (see Tables 1, 2). Then, representatives from each participating country (educational experts) were gathered in order to review the statements and give feedback. This was realised during visionary workshops with the representatives. The feedback was gathered and analysed in order to conclude with the final version of the statements in each one of the items in the three levels. The final version of the statements, the OSOS-SRT Questionnaire, is presented in Supplementary Annex S2. The OSOS-SRT was an online questionnaire, available for the schools that had access to the OSOS portal (https://portal.opendiscoveryspace.eu/en/osos/srt/).

The three levels of openness are considered to have equal contributions to the calculation of the total score. The weight of each one of the eight items, within each level, is also of the same importance (that is 0.125). The openness level of each school, which corresponds to the final score (openness level), is calculated as the mean of the scores that will achieve by selecting the respective statement in each one of the items (able to select one out of four statements).

Each one of the statements in each item corresponds to one of the subscales (Enabled, Consistent, Integrated, and Advanced), and when the responder is selecting the one that represents the status of the school unit, a specific number is appointed to that item. The subscales for each one of the items correspond to the following values, when the responder is choosing them: Enabled—25, Consistent—50, Integrated—75, Advanced—100.

As it was stated above, the minimum score that a school unit could achieve (or the starting point) is 25%. This would be the case of a school unit that would choose for all the items in all the levels the 1st statement that was corresponding to the Enabled School status.

Process and Sample

Each participating school had to fill in the OSOS-SRT instrument twice, initially at the start of the process and then after an academic year of implementation of the Open School Roadmap. The scope was to compare the level of openness at the beginning of the period and at the end. This would indicate that the proposed Open Schooling approach in the operation of the school unit is under study.

A total of 500 schools were involved in this study and provided data on their openness status in two measurements. These 500 schools were coming from 11 European countries (Greece, Germany, Italy, Portugal, Finland, Bulgaria, France, Germany, Netherlands, Ireland, and Spain) and represented both urban and rural schools. National and local educational authorities have organized open calls for participation. Following the schools’ expressions of interest, a representative sample of schools (primary, secondary) from urban, sub-urban, and rural locations and the schools were invited to participate in the study. The number of the schools in each country was balanced. Considering the complexity of the task, design, implementation, and support of numerous open schooling activities in the participating schools, a sample of 500 schools was considered the optimum choice from one hand to provide a significant amount of data for different school environments and on the other hand to support a realistic plan of implementation of the open schooling approach. Following the pre-measurement, we have studied the score distributions to meet the requirements of the normal distribution. School heads were supported to develop their localised open schooling plans and to adapt them into the relevant needs of their schools and the local communities. It has to be noted that the development of the plans is a process that has been introduced to European school communities involved in innovative projects for some years (EC, 2015b). These plans act as a reference point for the school’s development and are offering the opportunity to see the overall process adopted by school communities during their reform efforts. These localised plans took into consideration local challenges and opportunities, as well as national initiatives touching upon innovation in science education and RRI, and customised accordingly the various open schooling strategies that were described in the Open School Roadmap along with examples of successful practice. Innovation processes were discussed and explained. The role of the change agents was highlighted. Guidance and support were provided in the framework of workshops and webinars. Training materials were also available on the OSOS portal (www.openschools.eu). Examples of good practices, acting as reference points for whole-school interventions, were also available on the portal. Schools were encouraged to take part in networks with other schools and to develop common projects and initiatives (Mogren et al., 2019). Participation in a network encouraged interaction and provided schools with opportunities to enrich their practices and professional context through cooperation within and between schools, universities, science centres and museums, local industry and research institutions, collaborative reflection, development and evaluation of instruction, exchange of ideas, materials and experiences, quality development, cooperation between teachers, students, parents, science communicators, local entrepreneurs and researchers. The participating schools have created communities of practice to implement their innovative projects, involving numerous external stakeholders that progressively took an active role in the implemented projects. These schools have designed and implemented more than 1,000 innovative projects focusing on local challenges and problems and providing innovative ideas and solutions.

All statistical analyses were conducted with IBM SPSS statistics version 24.0 (IBM, Armonk, NY, United States). The central limit theorem is implied, and we assumed normal distribution because of the sample size of 500 schools was considered large enough (Wilcoxon, 2012; Field, 2013). For assessing the efficiency of the OSOS-SRT, a principal axis component analysis (PAF) was applied to extract the three factors. For comparison of measurements between the two periods, we used excel to produce histograms with normal curve to demonstrate the scores of the schools and the frequencies.

Results

In this paragraph, we present the results from the validation of the tool, using the factor analysis methodology as well as the results from the data gathered from the 500 participating schools.

Assessing the Efficiency of the OSOS-SRT

A factor analysis of all pre-test data (which were supposed to be “unspoiled” of involved interventions) verified the reliability and validity of the OSOS-SRT instrument. The hypothesized three-partite structure clearly appeared. As the factor analysis, the statistical method describes variabilities of different correlated items of a lower number of latent variables with the intent to reduce the number of the related variables; in our case, all introduced items of the tool can be reduced by applying certain general measures to three different unobserved and underlying variables, so-called factors. Typical measures are pre-defining min-eigen scores or identifying screen plots by following defined mathematical procedures.

We subjected all original 24 task-specific statements of OSOS-SRT to a principal axis factor analysis (PAF). The KaiserMeyerOlkin measurement of 0.934 is high, as is Bartlett’s test of sphericity (chi-square = 2,498.825 p =< 0.001) (Field, 2013). By using the KaiserGuttman criterion, 35.02% of the total variance was explained. Oblique and orthogonal rotations yielded essentially the same solution. The Varimax factor loadings are shown in Table 3. The extracted three factors covered perfectly the three hypothesized domains: management level, process level, and teachers’ professional development level. In total, our 24 task-specific statements solution ideally portrays the self-reflection tool on the sample basis of all participating schools. The reliability of the total scale scored 0.916, and any item omission would have decreased this score. On that basis, we desisted from omitting the two items with the only cross-loadings exceeding 0.4 (PL 7, PL 8) and continued the subsequent analyses on the basis of that structure to portray the school and its development.

TABLE 3

TABLE 3. Loading pattern of the OSOS-SRT with its three-actor solution and the corresponding items.

Monitoring Schools Development With OSOS-SRT

The data from 500 schools (from 11 different countries across EU) demonstrate that the Open Schooling approach (documented in the Open School Roadmap) had a significant impact on the openness of the participating schools. The Open Schooling approach (based on the OSOS project) has demonstrated an extremely successful journey in the diffusion of innovation in school settings. Recall how an Open School Culture requires schools, in cooperation with other stakeholders, to become agents of community well-being. In this framework, families are encouraged to become real partners in school life and activities; professionals from enterprises and civil and wider society are actively being involved in bringing real-life projects in the classroom. These projects developed by school forms successful networks with a taste for responsible innovation. The data demonstrate significant growth in openness (>11% on average) while the growth is much higher for less advanced schools (goes up to 45%) in a one-year intervention (Figure 1). The graph plots the post-values versus the pre-values for the 500 schools. To provide a prediction of the expected annual development for school leaders who are adopting the Open Schooling Roadmap and to provide suggestions on how school leaders and practitioners can employ the assessment tool in their administration, we are modelling the expected value as a function of the initial school score (Eq. 1). Clearly, there are many factors (e.g., size of the school) that could have a significant contribution to the final school score, but our intention in this study is to demonstrate the net contribution of the Open Schooling approach to the development of schools, independently of the local conditions and other factors that can affect the final result (see Limitations of the Study).

O p e n n e s s L e v e l (P o s t) = 36.5 \times {1.011}^{O p e n n e s s L e v e l (P r e)} - 0.06 \times O p e n n e s s L e v e l (P r e) . (1)

FIGURE 1

FIGURE 1. The graph presents the pre- and post-openness levels of the 500 schools that were involved in the study, after one academic year of implementation of the open schooling activities in their settings. The data (post- vs. pre-measurement) demonstrate significant growth in openness (>11% on average) while the growth is much higher for less advanced schools (goes up to 45%) in a one-year intervention. The fit line represents the average growth of the participating schools according to their pre-measurement.

Eq. 1 represents the expected openness level (post) of the school after one full academic year of interventions, with respect to its current openness level (pre) as it was measured with the OSOS-SRT instrument. The model provides an almost perfect match with the schools’ data (RMSE value very close to 0 and the R² of the regression is relatively high). It has to be noted that the exponential fit offers the opportunity to represent the significant increase of the schools which are scoring quite low at the pre-measurements, and their final scores are eventually much higher but at the same time to represent successfully the much smaller—still significant in the specific time frame—increase of the final scores of schools which were scoring quite high during the pre-measurements. The corrective term in the model ( $- 0.06 \times O p e n n e s s L e v e l (P r e)$ is used to improve the prediction of the model for schools that are scoring quite high at the first measurement. Table 4 presents the fit parameters.

TABLE 4

TABLE 4. Fit parameters.

Figure 2 illustrates the frequencies of the final scores of the sample (n = 500) with an indication of a normal distribution demonstrating the results pre- and post-measurement. We observe a significant increase of 11.34% of the mean openness value between the pre- and post-measurement, after 1 year of implementation of the open schooling approach in the school settings. The increase of 11.34% in the openness level that we observe indicates that schools integrated many of the aspects of the Open Schooling proposed approach in their day-to-day activities as well as into the Development Plans by reaching almost 67% openness value.

FIGURE 2

FIGURE 2. Frequencies of the measurements (pre and post) of the school sample. In the upper graph, we can see the results from the pre-measurement, before the schools start implementing the proposed open schooling approach while in the lower graph, we can see the results of the post-measurement, after one-year intervention. The Final Score indicates the average results (achievement) from the three levels of openness. The vertical line in each one of the graphs illustrates the mean value of the schools’ scores in each one of the periods.

The following figures illustrate the results in each one of the three levels (management—Figure 3, process—Figure 4, and professional development—Figure 5) for the schools participating in the sample (n = 500). We observe a balanced increase in all levels demonstrating the sensitivity of the tool to monitor the organisational change of the school environment. More specifically, the management level of the sample was increased by 11.92% while the process level by 11.65% and the teachers’ professional development level by 10.64%.

FIGURE 3

FIGURE 3. Frequencies of the school scores in the management level. In the graph are presented the results from the pre- and the post-measurements. The vertical line illustrates the mean value of school scores.

FIGURE 4

FIGURE 4. Frequencies of the school scores in the process level. In the graph are presented the results from the pre- and the post-measurements. The vertical line illustrates the mean value of school scores.

FIGURE 5

FIGURE 5. Frequencies of the school scores in the teachers’ professional development level. In the graph are presented the results from the pre and the post-measurements. The vertical line illustrates the mean value of school scores.

Discussion

According to Messick 1995 (p. 5), “validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions on the basis of test scores or other modes of assessment.” This definition is based on the theory that validity consists of one general form, construct validity, as it indicates just one overall evaluative judgment, rather than many. This singular view evolved from an earlier notion that validity consists of three independent types: criterion, content, and construct validities. Criterion validity refers to how well test scores correlate with real-world criteria, and predictive and concurrent validities are extensions of this broader concept. Typically, criterion validity relies on correlational analyses (Kane, 2001). Content validity assesses the degree to which a test covers and represents a domain of interest. This type of validity typically relies on expert judgment (Kane, 2001). Construct validity is typically defined as the extent to which a test accurately measures the construct of interest. Assessment of construct validity is not as straightforward as assessment of criterion or content validity. Construct validity is the degree to which a test measures what it claims, or purports, to be measuring. The proposed OSOS-SRT instrument shows a clear three subscales’ structure. Therefore, the operationalisation of the conceptual framework into a 24 task-specific statements for the school status succeeded to guarantee the content validity of the scale (Carmines and Zeller, 1979; Karabenick et al., 2007). As a newly developed instrument to monitor Open Schooling, the extracted factor analytic structure assures a robust framework in accordance with the hypothesised characteristics. For instance, quality measures are the fact that almost all items exceeded a loading score of 0.5 as proposed by Hair et al. (2010). Similarly, the high Cronbach alpha score of 0.916 for the total scale (as a most commonly measure used among reliability coefficients) points to a very high internal consistency (Cronbach, 1978; Green and Yang, 2009). As none of the scoring exceeds 0.95, we argue that no redundant items were in the scale (as such very high scores would provide an indication of this; Yang and Green, 2011). Thus, the extracted three factors covered perfectly the three hypothesized domains: management level, process level, and teachers’ professional development level. It proved that it was sensitive to monitor the extremely successful journey in the diffusion of innovation in school settings in the specific timeframe of one academic year.

In consequence, by turning the argument away from the scale’s quality to the targets, Open Schooling works, when introduced in numerous schools from different countries. It can monitor transforming schools (that are following a well-documented and systematic path as described in the Open School Roadmap) into innovation hubs within local communities by supporting student projects to introduce real needs into a community outside of school, present them publicly, and enrich local expertise (OECD 2020a). Due to the valid instrument availability (see above), we can point to a broad improvement potential of Open Schooling incentives. The described instrument enables a quick testing of the self-reflection process which is regarded as a powerful way to facilitate the school development (OECD, 2013; EC, 2019b). Thus, OSOS-SRT seems sensitive enough to communicate evidence of the school development progress and provide insights to teachers, school leaders, policy makers, and parents. More specifically, the participating schools demonstrate an increase at the management level of 11.92%, at the process level by 11.65%, and at the teachers’ professional development level by 10.64%. The data demonstrate an increase that goes up to 35–45% for schools that have lower openness level at initial assessment. Still, schools that succeed higher scores in the first measurement demonstrate an increase in their final scores. Although the time interval of the intervention is only one academic year, the OSOS-SRT seems to be capable to identify the schools’ progress. These results are in line with the basic principles of the open school environment. An open school is a curious, welcoming, democratic environment that supports the development of innovative and creative projects and educational activities (EC, 2017b). It is an environment to facilitate the process for envisioning, managing, and monitoring change in school settings by providing a simple and flexible structure to follow, so school leaders and teachers can innovate appropriately for school local needs (Earley and Greany, 2017). Our results are also in line with the expectations of the EU Policy document “Science Education for Responsible Citizenship” (EC, 2015a) where collaboration between formal, non-formal, and informal educational is earmarked to enhance relevant and meaningful engagement of all societal actors with science and increase uptake of science studies and science-based careers to improve employability and competitiveness.

Based on the analysis, we are proposing to apply our model to predict the school development level after one full year of adoption of the open schooling approach, especially as the literature about engaging actors outside the academy is still limited (EC, 2015c). The model is described by Eq. 1 and predicts the expected openness level (post)—after 1 year of intervention—for a school that has currently an openness level (pre). The model is based on the fit that was applied to the data from the 500 schools that were analysed in the framework of the study. We have adopted the model with the lower RMSE value (0.1764) that indicates an almost perfect match between the data and the predicted value (Hyndman and Koehler, 2006; Willmott and Matsuura, 2006; Pontius et al., 2008). This process highlights that one of the core elements of the schools’ success in raising achievement is a robust focus on tracking and monitoring of development progress and the use of assessment data for progress tracking, target setting, and support for schools slipping behind with targeted interventions. Such models can be used effectively by senior managers, teachers, teaching assistants, and governors to answer questions about current standards, trends over time, progress made by the schools and to set high expectations in case study schools (Demie, 2013).

Open school environments may foster learner independence—and interdependence—through collaboration and through providing opportunities for teachers to collaborate with colleagues and external stakeholders and to form communities for exchanges of good practices. In such a setting, being part of the community is becoming a unique professional development opportunity. The open school environment facilitates the cooperation with external experts who can share their experiences and knowledge with the teachers. In that way, teachers are becoming confident and more efficient in their educational tasks. These interactions and cooperation could result in significant learning outcomes (Sotiriou et al., 2017b; OECD 2019). The significant increase in the teachers’ professional development scores in the participating schools is the outcome of this process.

Finally, the adaptation of specific activities entails linking their subjects to issues of national interest in connection with the grand challenges of our times. Schools thus aim to “act locally but think globally,” a motto developed a decade ago (Hopkins, 2011) now but still far from the reality of most schools in Europe today. For example, the aim of the Schools Networks Alert Citizens Protection initiative, a pilot project for the open schools, was to a) transforms schools in south-eastern Mediterranean basin countries into local hubs of education, innovation, and information about earthquakes and disaster prevention, connecting them with local authorities, local civilian protection agencies, local business, research and science centres, and other local stakeholders in the process and b) engages students in real-life projects that are proposing innovative solutions adapted to the local conditions by employing real-problem solving skills, handling and studying situations, and participating in meaningful and motivating science inquiry activities on earthquake disaster prevention and mitigation. In this way, these schools, by involving numerous stakeholders in the school initiatives, enrich the science capital of the local communities and promote responsible citizenship (e.g., Watkins 2005; Philips, 2006; Howard, 2012; Okada et al., 2019). Such an innovation programme holds great potential. If we want a powerful, innovative, and open culture in schools that is self-sustaining, we have to empower system-aware practitioners to create it, whilst avoiding simply creating interesting but isolated pockets of experimentation. The open school processes instil a design-based approach of collaborative learning and inquiry between professional practitioners, thus creating a “pull” rather than “push” approach. To promote such an approach in the current schooling practices, an ecosystemic standpoint should be taken from the side of the remedying initiatives. More specifically, the latter should aim to capture the profiles, needs, contributions, and relationships of all these school-related actors and elements towards a sustainable innovation ecosystem (UNESCO, 2014; Oksanen and Hautamaki, 2015) that operates under a holistic framework of organizational learning and promotion of educational innovations (Senge et al., 2012). This process was implemented in the participating schools. The school communities have introduced innovative practices in their settings by involving the whole school community. The necessary adoptions were made to the curriculum in order to facilitate the development of students’ projects.

Applying the open schooling approach in local settings has made it clear that schools have much to gain by fostering connections between formal and informal learning, between existing providers of education and new entrants (Nghia, 2018). The open schooling approach has supported the transformation of schools into open schooling environments in different countries. A tailored package of supporting materials, including the OSOS-SRT and the Open School Roadmap, has been developed to support schools as they transform into Open Schooling Hubs, offering a clear mentoring approach to schools with a vision for the future. Guidance has also been provided to local and system-level stakeholders. The described instrument exemplifies the overall approach on how we can best support schools in their attempt to evolve, transform, and reinvent their structures towards a more open, localized, and socially responsible learning environment. In this framework, schools facilitate open, more effective, and efficient co-design, co-creation, and use of educational content (both from formal and informal providers), tools, and services for personalized science learning and teaching that from the basic ingredients for innovative student projects. Such projects understood as best practices are the so-called incubators and accelerators on school innovation. And what is more: Its suitability to heterogeneous school systems (11 EU countries with 11 different systems) reassures our way to open up schools. Practitioners and school heads could advance the expected learning outcomes in the framework of such activities as they are expected to increase the motivation and interest of the students. This work differs from the work in engagement and affect in terms of timescale. Whereas engagement and affect often manifest in brief time periods—as short as few seconds—motivation and interest are more long-term stable aspects of students’ experience. Work by Kizilcec et al. (2017) has tried to connect students learning experience with their values, leading to greater degrees of completion of school projects and related tasks. Walkington and Bernacki (2020) have tried to implement projects related to students’ personal interests, leading students to work faster, become disengaged less often, and learn more. Furthermore, the overall self-reflection process could help leaders to reflect on the ways they are/are not facilitating the proposed innovation framework through both their individual actions and the systems and structures they work to create and uphold. The OSOS-SRT tool is intended to support reflection, rather than evaluation or accountability purposes. Reflecting on one’s strengths and limitations and making a plan for ways to improve and grow leads to improvement. This tool is intended to support that reflection process. Within any system, education included, there are elements within our circle of control that one can directly address, elements within our circle of influence, and elements that are within our circle of concern. This reflection tool focuses on a school leader’s circle of control and their circle of influence. Although the categories and reflection statements of the tool are designed to cover many elements of school openness and innovation potential, we acknowledge that there are areas where one should “dig deeper.” This tool is not intended to stand alone, but rather be used in conjunction with other methods of reflection, ongoing coaching, professional development, and feedback.

Potential Limitations

After the instrument’s validation (which we assured by analysing all unspoiled pre-test data) and its subsequent application to monitor program effects during a school year, a follow-up study with an unspoiled control sample must be the next step. Although conditions for unspoiled frames will be difficult to define as every school year is a traditional but also complex entrepreneurship, this piece currently is missing in the portrayal. Nevertheless, as different school management may produce a different outcome, the SRT may provide a “thermometer” for comparison offering prescriptions for different school backgrounds, school types, or governance models. Within this context, it needs consideration that the overall study approach has followed the model adopted by large-scale surveys like PISA (OECD, 2020b) where the reporting of school data and development is based on the contributions and the views of school heads who are considered the key stakeholders in school development-related studies. Additionally, Eq. 1 that represents the post-score of the school as a function of the pre-value has been developed as a way to demonstrate to school leaders and practitioners the overall openness growth provided that the Open Schooling approach was fully adopted. The fact is that there are many factors that could affect the final result. The size of the school is a very important factor. Such schooling environments require major effort to embark on the transformation journey. The training of the staff is time-consuming, and the overall system cannot achieve high transformational momentum in 1 year. There are also other parameters like the interest of the local communities and their involvement in the school practices and activities. The national educational policies are also key factors that could affect the schools’ transformation journey. The current study is offering an initial glimpse of what can be achieved in the schools having a clear action plan, with well-defined objectives and a localized strategy that meets their needs. In the framework of the study, this work was done for every single school that was involved. The results of the study could be considered as a reference point for schools’ communities that are interested to design their own path towards openness.

Conclusion

Improving and modernizing education and training systems need support by assessment in order to fine-tune the process. The described instrument has been shown its validity to enable school heads and innovative teachers to follow up their progress and to get the opportunity to reflect. Using this self-assessment as a starting point, school heads can identify areas for growth as leaders and can give further thought and planning to the development of their plans. The purpose of this tool is to help to develop reflective practitioners who can lead schools towards the transformation of schools to innovation ecosystems. There is a considerable need to support and involve a broad set of stakeholders (e.g., school heads, teachers, administrative staff, students, and external stakeholders) so that each school can respond to the need for institutional change and development in a meaningful, comprehensive, and strategic way. This is becoming even more critical and important in unexpected situations like the current pandemic that has posed numerous restrictions to the schools’ operation. In such cases, the schools’ openness and autonomy offer a variety of processes and methods to cope with the rather demanding framework of operation during the school’s closure. The self-reflection process and the schools’ openness could act as drivers for school communities to adopt innovative methods and keep delivering high-quality educational services to their students. The described OSOS-SRT could act as the map in such a transformation journey.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Authors Contributions

MS analysed and interpreted the data regarding the openness level of the schools as well as had a major contribution to the process of assessing schools’ openness, the methodology and contributed to the discussion. SS developed the introduction part explaining the framework for the open schooling as well as was a major contributor to the process of assessing schools’ openness, to the methodology and the discussion. FB analysed the data to provide the factor analysis for the effectiveness of the OSOS-SRT as well as contributed to the discussion. All authors read and approved the final manuscript.

Funding

This research was supported by the OSOS-Project (European Union Grant Agreement, No.741572) as well as the University of Bayreuth granted financial support. The funders had no role in the design of the study, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results. This article reflects only the authors’ views. The funders are not liable for any use that might be made of the information contained herein.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We appreciate the support of the school heads and teachers of the 500 schools that were involved in the study and the share the status of their schools with the research team.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2021.714227/full#supplementary-material

Abbreviations

OSOS, Open Schools for Open Societies; OSOS-SRT, Open Schools for Open Societies Self-Reflection Tool; RRI, Responsible Research and Innovation; CPD, Continues Professional Development; ML, Management Level; PL, Process Level; TPDL, Teachers’ Professional Development Level.

References

Biesta, G. (2009). Good Education in an Age of Measurement: on the Need to Reconnect with the Question of Purpose in Education. Educ. Asse Eval. Acc. 21, 33–46. doi:10.1007/s11092-008-9064-9