Data in Schools: (Changing) Practices and Blind Spots at a Glance

Krein, Ulrike; Schiefner-Rohs, Mandy

doi:10.3389/feduc.2021.672666

REVIEW article

Front. Educ., 13 December 2021

Sec. Leadership in Education

Volume 6 - 2021 | https://doi.org/10.3389/feduc.2021.672666

Data in Schools: (Changing) Practices and Blind Spots at a Glance

Ulrike Krein*

Mandy Schiefner-Rohs

Department of Educational Sciences, University of Kaiserslautern, Kaiserslautern, Germany

This review aims to provide a concise overview of the role of (digital) data and new data practices in schools. By focusing on the impact of data on pedagogical practices, it aims to shed light on how the everyday tasks of teachers and other pedagogical staff in schools are changing, particularly as a result of the generation and use of digital data. For this purpose, existing studies and previous theoretical debates on this topic are examined for their perspectives on data and data practices in schools. The pedagogical data practices of (improving) teaching and learning, assessment and counseling, (data-driven) decision-making, and cooperation and collaboration by “doing data” will be elaborated and discussed. Likewise, data practices that are missing from the studies are identified. We conclude with an overview of blind spots and further research needs.

1 Introduction

Demands for governance and control in the education system have been increasing worldwide (see, e.g., Mandinach and Miskell, 2017; Brown et al., 2017; Park et al., 2017; Schildkamp, 2019; Datnow, 2020; for Germany Altrichter and Maag Merki, 2010; Ackeren et al., 2015). Thus, it is not surprising that a governance perspective has been widespread in the school context for years and is gaining importance in the international discourse about the effectiveness of schools. Along with such mostly numerical governance perspective comes the increased emergence of standardized system-level metrics, as well as the growing production of data as a central part of school governance processes in schools themselves (Selwyn and Pangrazio, 2018). This usually involves the promise of being able to know more about schools and schooling through data (Selwyn et al., 2021). While it has always been common practice in schools to collect data and transform student performance into data, this process has not been very systematic and certainly not routinized or even automated (cf. Mandinach, 2012, p. 72). “But now there are technologies that can assist them in this process (ibid.). There is no question that data now are proliferating, and new sources of data continue to emerge. The pressure on educators to use data is increasing” (ibid.). The article explores these perspectives by examining the role of mainly digitally generated data in pedagogical practices in schools and how this manifests itself in (maybe new) data practices. In doing so, we will focus on the pedagogical practices of teachers and other educational staff in schools from a phenomenological perspective. Well aware of the existing and widespread discourses around the use of data, for example, on school effectiveness (e.g., Hopkins et al., 2014; Reynolds et al., 2014) or in the context of school leadership (e.g., Levin and Datnow, 2012; Bowers et al., 2014; Schildkamp et al., 2019), the influence of data on pedagogical practices in schools seems to play only a minor role in the theoretical as well as the empirical debate so far. In the following article, we are therefore not interested in examining data-driven decision-making in schools, for example, for school development as a heuristic or process, but rather in looking at whether and how digital data and the rationalizations it reveals are also inscribed in pedagogical practices. In order to gain initial insights into findings on pedagogical data practices, on the one hand, and to map implications for future research, on the other hand, this article provides an overview of the theoretical and empirical discussion of pedagogical data practices and data use in schools to date. To do so, the following section first describes our understanding of data and pedagogical practices, before our methodological approach used in the literature review is clarified. Synthesizing and describing the findings is followed next. Besides describing previous research perspectives on the topic and pedagogical practices that have been examined so far, previous blind spots in light of the literature reviewed are also discussed. The article ends with a reflection on the limitations of the article and future challenges for further research and schools.

2 Theoretical Framework

2.1 Understanding Data

In parallel to the educational programmatic strengthening of the generation of data from a governance perspective, other starting points can be found that spur the creation of digital data in schools: For example, the generally advancing digitization of schools plays a crucial role. On the one hand, it is possible to generate more data to monitor student performance; on the other hand, in many cases, this data generation is done unconsciously through the use of technology alone (e.g., Selwyn et al., 2015). Thus, through the mere use of digital technologies, we all generate data and data traces in the background, which can then be fastly and easily evaluated and interpreted (Breiter and Hepp, 2018). For example, through the use of learning platforms, digital traces or metadata are recorded, which can, for example, make login times or the number of correctly completed tasks of individual students visible (ibid.). Schools in particular, as places of formal education, rely on translating teaching and learning into data—just think of the assessment or grading of students. At the same time, the use of data under the terms “learning analytics” and “educational data mining” also promises to reduce the workload of teachers and is “often promoted enthusiastically as an innovative means of enhancing teaching and learning” (Selwyn et al., 2021, p. 2; see also; Dander and Aßmann, 2015). Related to this are also educational policies and pedagogical imperatives for school actors (Selwyn, 2020) to proactively use data for their own actions: “Key requirements here include the capacity to ‘notice’, ‘interpret,’ and ‘construct implications’ from data […]” (ibid., p. 2). Data practices thus become part of daily school in their digital ecosystems (in the sense of permeating school with technology) or part of individual practice (Decuypere, 2021). At this point, in addition to a governance perspective through data, a second perspective on data in schools thus opens up, namely, the collection of data for pedagogical purposes, mostly within the classroom. Teachers’ data practices oscillate between “tracking data” and “keeping data on-track” (Lewis and Hardy, 2017) as actively constituted forms of governance. Used under the assumption that technologies or algorithms are supposedly neutral and overcome the subjectivity of, for example, teacher assessments, more and more data points are generated and used as appropriate. However, what usually remains unnoticed here is that the actions that produce data are interwoven within the structure and materiality of the technology itself (Allert et al., 2017). It is also unclear how this interweaving of data and (pedagogical) actions in schools occurs, to what extent data influence existing pedagogical practices, or how these practices change or are newly constituted through the generation and use of data. Although—and this is what is special about data and their generation and use not only in school—data are not neutral (as Kranzberg, 1986, p. 545 expressed “neither good nor bad; nor is it neutral”), it can be assumed that they enter constitutively into pedagogical action and possess normative affordances (Allert et al., 2018). It seems reasonable to assume that pedagogical practices are also affected by this phenomenon. Since pedagogy has also always faced the challenge that pedagogical, especially learning, processes are “invisible” per se, i.e., they always need to be operationalized.

2.2 Data as Part of Pedagogical Practices

In order to examine pedagogical practices in schools, it is necessary to first clarify the perspective on practices as the object of study, both from the perspective of pedagogical practices and from the perspective of resulting data practices related to school. According to Reckwitz (2003), social practices can be generally understood as the smallest unit of social life as “know how dependent behavioral routines held together by a practical ‘understanding,’ whose knowledge is, on the one hand, ‘incorporated’ in the bodies of the acting subjects, which, on the other hand, regularly take the form of routinized relations between subjects and the material artifacts they ‘use’” (ibid., p. 289; authors’ translation). Pedagogical practices, in turn, are mostly dispersed practices (Schatzki, 1996) and are found especially in schools. Such practices—like questioning, ordering, and describing (ibid.)—become visible within many practices because they are interwoven into other practices and nexuses of practices (cf. Heidenstrøm, 2021, p. 5). “They are dispersed because they only exist within other practices, and people are usually engaged in an integrative practice when they carry out a dispersed practice” (Schatzki, 1996, p. 99). In school, dispersed practices are evident, such as educating, learning, teaching, assessing, caring, practicing, counseling and, organizing (see also Kade et al., 2011), which are also interwoven with each other. Data practices subsequent to this can first be understood in the general sense of the word as actions, performances, and resulting consequences that bring about the introduction of data-producing technologies into everyday educational situations, which means everything that is “done” in schools are in, with, and through “data.” We therefore follow the reading that data practices are constituent elements of social life: This broadens the perspective to the different ways in which data are “brought to life,” how actors relate to and begin to work with data. At this point, it is essential for us to shift the perspective away from the level of the provision of technology to socio-material practices (cf. Allert and Richter, 2017). Understanding practices with reference to socio-materiality means understanding technologies “not as compliant and neutral tools that do their job in the hands of teachers and learners without further ado” (Röhl, 2013, p. 2), but rather taking technologies seriously in their constitutive participation in practice (cf. ibid.). It is therefore relevant for us to look at “doing data” (ibid.) in schools and the reciprocal influence of data practices on actors’ ways of thinking and acting or enabling them. In this context, Marres and Stark (2020) emphasize that data points and the conducted activities themselves are also not necessarily distinct: “increasingly, the conduct of educational activities—and of social life more broadly—coincides with the direct and continuous (‘live’ and ‘real time’) capturing of these activities as data points” (Decuypere, 2021, p. 68). When considering data practices, it is therefore important to consider not only the interconnectedness of media and action but also the interwoven nature of data and practices themselves. An increasingly interwoven or “co-” impact of media, data, and action has been internationally observed in the school context for several years and has already been highlighted in research (ibid.). For example, Williamson (2020) recently points to the desire for the commodification of education and sociality through digital education platforms, but also the control and monitoring of student activities and their digital data traces have become common practices (Lewis and Hardy, 2017). Data practices, however, are also becoming increasingly important in formal, non-formal, and informal learning contexts (Decuypere, 2021) including, related educational apps (Decuypere, 2019b) and behavioral class management software (Williamson, 2017; Manolev et al., 2019). But, “To research data infrastructure is to research the development practices that are, even as we survey them as ‘objects’ of study, reconfiguring the very conditions of this study; data infrastructure is not simply a set of material supports ‘out there’; it is an assemblage of practices” (Gulson and Sellar, 2019, p. 773). Against the background of these theoretical assumptions our question therefore is how are pedagogical practices influenced by data?

3 Methodology

In order to provide an overview of the state of research to date, a search of published studies and theoretical contributions dealing with data practices in schools was conducted loosely based on Newman and Gough (2020). For this purpose, publications were reviewed in which the use of data or datafication in the school context was considered theoretically or empirically. The methodological approach was multi-step (Figure 1).

FIGURE 1

FIGURE 1. Methodological Approach.

In a first step, relevant databases such as FIS Education, SAGE Journals, ERIC, and SCOPUS were consulted in the second half of the year 2020 in order to represent a broad spectrum of published studies.¹ The goal was to find articles dealing with the use of data or datafication by teachers in schools. For this purpose, defined search terms as “data,” “datafication,” “algorithm,” “practices,” and “school” were used with regard to titles, abstracts, and keywords. These were combined with appropriate operators (AND/OR) in several rounds of searches. All results since 2009 were considered. The time restriction was chosen on the one hand because of the growing importance of digital data in various contexts in recent years and the consequent increasing research-side engagement with data as a research object. On the other hand, this was intended to provide an overview of the current state of research. In addition to a publication date after 2009, the following inclusion criteria were used:

• The publications examine how teachers and other pedagogical staff use (digital) data in pedagogical school contexts.

• The publications are empirical (qualitative, quantitative, and mixed method) or theoretical in nature.

• The publications are available in German or English.

• The publications are freely accessible or could be retrieved through institutional access.

The search identified several thousand results, in which abstracts were screened in a first step for their fit to the criteria mentioned before. Since the goal was to review studies that were applied in the school context or related to schools, all articles that reported on other perspectives or focal points, such as institutions of higher education or the commercial promotion of individual digital tools, were excluded. In addition, due to the focus on pedagogical practices of teachers and other pedagogical staff with data, publications addressing school effectiveness research, school leadership research, policy makers, or the meta-level of schools were also excluded. Consequently, all those German and English language publications were included in the analysis that were published between 2009 and the first half of 2021 (time of the second search) and had a thematic focus on the use of data by teachers and other pedagogical staff in schools. To ensure reliable screening, publications were reviewed by multiple researchers. Disagreements regarding fit were resolved by consensus. To complement the retrieval following snowballing (Wohlin, 2014), additional searches were carried out in the reference lists of the selected datasets as well as in relevant publications that had recently been published (e.g., Decuypere, 2021). A total of 39 publications dealing with data, datafication, and pedagogical practices in school contexts were included in the present review in this first phase.

In a second step, further literature and author references were obtained from experts in the research field, and based on this, a supplementary search was conducted in order to close any remaining gaps in the data corpus. The 167 publications found here were then also reviewed with regard to the inclusion criteria listed before and checked for fit. As mentioned before, a consensual screening was conducted, after which 28 more publications could be tapped. In total, the data corpus of the present article comprises N = 67 publications.

To obtain an overview of the state of research to date, the next step was to categorize and analyze each article in detail with regard to 1) the research questions pursued, 2) the methodological approach (data collection and analysis), 3) the target group addressed, 4) the results and main findings, 5) the data practices studied or described, and 6) the perspective taken. A narrative approach (Popay et al., 2006; Snilstveit et al., 2012) was chosen to synthesize the findings and implications of included studies due to the exploratory nature of our research. In this way, we were able to identify and trace highlights and trends in the discourse of data and data practices in school contexts.

4 Findings

Based on the literature reviewed, the following will provide an initial overview of the pedagogical data practices visible in studies and the related perspectives on data, data practices, and those involved in them. For this purpose, the individual perspective approaches to data, data practices, and also the addressed target groups from the included articles will first be presented in order to reveal the phenomenology of data practices in the school context in the preceding debate.

4.1 Phenomenological Perspectives on Data in School

In the previous discussion of (digital) data in a school context, two different perspectives on data can be identified: First, data are mostly understood as numerical generated data (Dander, 2018a) that possess a certain character as a “stable, fixed entity” (Selwyn, 2020, p. 3). Data can be classified into “four main categories of data: 1) provided data; 2) observed data; 3) derived data; and 4) inferred data” (see OECD, 2014, p. 5). In this context, data are negotiated as an objective representation of truth and are considered as “any information that helps educators know more about their students and can be codified in some way” (Jimerson and Wayman, 2015, p. 3). However, there are different perceptions of how the relationship between data and information is shaping up. So Schildkamp and Kuiper (2010), p. 482 pointed out that “data can be transferred into information by, for example, contextualizing, categorizing, calculating, connecting, and summarizing the data (Davenport and Prusak, 1998). In order for data to be used, the data should be transferred into information by interpreting the data” (see also Spillane, 2012; Abrams et al., 2016; Espin et al., 2017; Vanlommel and Schildkamp, 2019). So “data (in a school’s context) are defined as ‘information that is systematically collected and organized to represent some aspect of schooling,’ for example, assessment data, structured observation data, and student survey results” (Prenger and Schildkamp, 2018, p. 734; also Ebbeler et al., 2016; Vanlommel and Schildkamp, 2018; Schildkamp et al., 2017; Schildkamp et al., 2015). Using data then implies interpreting it and so transforming it into information (cf. Schildkamp and Kuiper, 2010, p. 482). Also, Mandinach (2012) refers that “data, in and of themselves, are meaningless. They are simply numbers” (p. 73), Waymann et al. (2012) call them “raw inputs” (p. 4) and continues “(e)ffective data use requires going beyond the numbers and their statistical properties to make meaning of them” (ibid.). In the existing studies, such a perspective of data between information and numbers in schools can be seen a lot, for example, in efforts to optimize learning processes of students: on the basis of login data and digital traces that students leave on learning platforms, attempts are made to trace learning processes and to make pedagogical decisions (e.g., adjustment of teaching methodology) based on this (e.g., Williamson, 2017). With such a view, the collection and use of data appears intentional, data here are “always to be given for a purpose […]: for their later interpretation, for their computer-technical evaluation, for later access, and for the understanding of reality (Gießmann and Burkhardt, 2014, p. 3, authors’ translation). Pangrazio and Selwyn (2018) make a distinction between personal data as 1) data that users give to devices/systems, 2) data that devices/systems extract from users, and 3) data that devices/systems process on behalf of users (cf. pp. 421–422). Bowers however focuses on data as grades, when he (2009) outlines that “schools are inundated with data, including grades, attendance, discipline records, and standardized test scores (Creighton, 2001; Wayman, 2005)” (p. 609). So, to sum it up with Schildkamp and Kuiper, “data use had become a part of the school’s culture, ‘the way we do things around here.’ Teachers did not think about data as ‘something that is done to the school’ but as something that ‘is done by and for the school’” (Schildkamp and Kuiper, 2010, p. 486). This is often linked to the assumption that data can help remove politics and ideology from decisions, so teachers can focus on teaching and learning (ibid.). Data then are used here as an objective means of reducing and correcting bias in judgment processes (Vanlommel and Schildkamp, 2018). Nevertheless, it should be kept in mind, as Schildkmap and Kuiper (2010) point out, “Data can highlight strengths and weaknesses, but it does not improve student learning in itself” (p. 486; also Park et al., 2017).

A second approach is to collect and understand data from a relational perspective in the social context of the school (Selwyn, 2020). The focus here is understanding data as dynamic entities that are negotiated processually by the respective actors and are therefore also contoured by their narratives when using and obtaining data themselves (ibid.). Thus, data there are not static but, to a certain extent, have “life cycles” (ibid., p. 3) that must be considered holistically when acting with data. This is especially the case for digitally collected data. The emphasis here is on the “co”impact of technologies; thus, the focus is not on digitized devices such as laptops, smartphones, or tablets and other tools but on the linkage and interplay of different actors with technology(/-ies) in order to address practices. Digital data are not reduced to mere tools but are seen as actors that enable and constrain behavior (cf. Kalthoff et al., 2016). The focus here is on ontological implications of data and how data shape reality and how we perceive the world through their ideological nature and the way they convey information (Selwyn et al., 2021). In other words, the focus is on “what data does within the social context of a school, as well as simply what data is” (ibid., p. 4).

4.2 (Pedagogical-Oriented) Data Practices in Schools

In this section, we will highlight different pedagogical practices and their intricacy with data found in the literature. As said before, our goal is to examine whether and how digital data and the rationalizations they reveal are also inscribed in pedagogical practices. Therefore, we identified common school practices and analyzed how they are currently interwoven with data and technology. The articles examined provided information about various pedagogical practices, although this was not the primary object of inquiry in all articles. For example, practices of teaching and assessment became apparent through the descriptions of the implementation of a digital instructional tool, although consideration of these practices was not the focus of the article (see, e.g., Zupanc et al., 2009; Hershkovitz, 2015). The analysis identified the following dispersed pedagogical practices that are interwoven with data and the process of technology-driven datafication: (improving) teaching and learning, assessing and counseling, and—overarching the others—(data-based) decision-making. Although these practices are interwoven, we will first look at them separately from an analytical perspective. In the following, these pedagogical practices with their connections to technologies and data are outlined before being discussed with a view to blind spots and implications for research on data practices in schools.

4.2.1 (Improving) Teaching and Learning

The vast majority of the contributions included have a focus on and can be attributed to practices for improving teaching and learning. Two perspectives can be distinguished: the use of data by teachers and the use of data by students.

On the teacher side, practices of instructional adjustment based on data are most apparent. Here, teachers can adapt their teaching methodology by using data (e.g., Schildkamp et al., 2017), such as homeworks, student surveys, and feedbacks (ibid.; Schildkamp and Kuiper, 2010; Kennedy and Datnow, 2011) and thus also change practices of lesson preparation and implementation (e.g., ibid.; Mandinach, 2012; Waymann et al., 2012; Prenger and Schildkamp, 2018). Dam and others (2020) describe an example in which teachers collect data on newly created lessons and compare them to standards or their previous lessons. Based on these results, they modify their teaching practices or “think about new ways to diversify instruction in the classroom” (Datnow et al., 2012, p. 263). Similarly, teachers have used formative assessments they developed to generate data to align with standardized tests and standards to adjust instructional plans when standards were not met (Sun et al., 2016). Zupanc et al. (2009) also report on a data-based tool (ALAT) for diagnosing strengths and weaknesses in learning approaches for instructional improvement, which enables joint problem analysis and discussion of one’s teaching approaches and thus stimulates improvement in educational quality. “In sum, using multiple forms of data to inform instructional decision-making can help provide a more holistic picture of student achievement and provide more ways to see evidence of student growth” (Datnow and Park, 2018, p. 141).

It is also evident in the literature that improving teaching and learning is often related to aspects of learning analytics and educational data mining and therefore also technology (for blended learning, see Mandinach and Miskell, 2017). However, despite the similar thematic focus, the articles differ in how data are obtained here: For example, while Mertala (2020) refers to a “hidden curriculum” and thus an all-encompassing data collection in the classroom, other authors describe, for example, real-time capture of actions in learning resources (e.g., Thompson and Cook, 2017; Brun et al., 2019) or data collection through physiological arousal (measured with a wrist-worn sensor) as a predictor for an intervention to measure engagement (e.g., Hershkovitz, 2015). Regardless of the type of data production, however, learning analytics is primarily seen as a helpful tool for teachers and learners in the learning process (Hershkovitz, 2015; Roll and Winne, 2015; Mandinach and Miskell, 2017; Ebner, 2019): by collecting and comparing data and creating patterns based on the data obtained, activities can be compared and learning progress can be made visible to all (of course, as many authors emphasize, taking data protection and privacy into account). In this context, adaptivity also emerges as a topic in the discussion about improving teaching and learning (Petersen et al., 2017). Many authors therefore also deal with the personalization of learning (“to teach,” “to learn,” “to belong”) through learning analytics and educational data mining, whether from a critical perspective such as Thompson and Cook (2017) or from the perspective of general feasibility and thus the improvement teaching and learning in school (cf. Ebner, 2019). Teachers use data or the results of their processing, as a basis for interventions to intensify the high level of individuality in the learning process and to provide targeted help to learners (e.g., Sun et al., 2016; Ebner, 2019); for example, learner profiles are determined from various data and student characteristics and, on the basis of these, instruction and teaching methodology are adapted. Personalized recommendations are made, and the efficiency and effectiveness of the learning process are increased through the associated quantifiability of the learning process (Williamson, 2017). In this context, the relevance of “longitudinal data system with information on previous test performance, portfolio grades, teacher’s comments on progress reports, and demographic data as being potentially powerful tools to tailor instruction to meet the needs of individual students” (Abrams et al., 2016, p. 23) was also emphasized.

Furthermore, data can also be used by students themselves to empower students to actively engage with the data generated in school (see Selwyn, 2020); Roll and Winne (2015) reported that on an environment in which it is possible to analyze learning behavior by interpreting and evaluating action sequences. Through the recorded tendencies of the learners, they can receive feedback and respond to it. This enables students themselves to adjust their learning process, to foster self-regulated and directed learning and may motivate them as well (Schildkamp and Kuiper, 2010; Mandinach, 2012; Sun et al., 2016). Jimerson and Reames (2015); Jimerson et al. (2016); Jimerson et al. (2018) as well as Kennedy and Datnow (2011) reported on the practice of student-involved data use (SIDU) in which “students are tasked with tracking and analyzing data about their own performance” (2018, p. 1). In doing so, the practice has different aims to improve teaching and learning: improving the instruction of teachers, improving student understanding, and to reflect on aspects of assessments. But it offers a dark side too: due to the mass of data that is requested, the participating students feel like sheer “numbers” and are no longer perceived as individuals (Roberts-Holmes and Bradbury, 2016). This offers ethical debates about subjectivation practices in schools.

4.2.2 Assessing and Counseling

Closely related to data practices of improving teaching and learning are practices of section 4.2.2 by the use of data. Here, data provide tools for teachers to assess and predict student performance (Selwyn, 2020). Such assessment practices are already apparent when students enter school; for example, students’ school readiness is assessed in kindergarten using a variety of data (Datnow and Park, 2018). Thus, the analysis of student data such as grades, dropout rates, or achievement standards and goals can open up a secondary level of formative assessment as it combines practices of summative and formative assessment (Zupanc et al., 2009; also e.g., Schildkamp and Kuiper, 2010; Datnow and Park, 2018). The results of these assessment practices are then in turn intended to inform curriculum planning or systemic school improvement (ibid.). Similarly, in addition to such overt assessment practices, data-based assessment practices are also found to occur implicitly or hidden in the context of (equally data-based) teaching and learning practices and, moreover, are not necessarily done by teachers; for example, a study by Mendiburo et al. (2013) points out that an algorithm first assesses students on their daily prior knowledge and progress based on various scores before ultimately grouping them (ibid.). Accordingly, the results of this data-based assessment practice provide the basis for further (data-based) pedagogical practices—even if these are carried out by non-pedagogical actors. Unfortunately, “(p)roblematic practices of tracking and ability grouping with long-term consequences continue to abound in schools and are legitimated with data” (Prenger and Schildkamp, 2018, p.148). Thus, critical data practices of assessment and grouping, namely, educational triage practices (Schildkamp and Kuiper, 2010; Park and Datnow, 2017; Datnow and Park, 2018), are also evident. Here, attempts are made to improve test scores by focusing on individual student groups (“suitable cases,” Schildkamp and Kuiper, 2010, p. 485), whereas weak students are neglected.²

Furthermore, there are several references in the literature to counseling data practices based on assessment results. In Australian studies, for example, data play an important role in the context of school career counseling (Selwyn et al., 2015; Selwyn and Pangrazio, 2018; Selwyn, 2020): in the schools described here, performance data such as Victorian Certificate of Education examination results were scaled (sometimes using Excel or Google Sheets alone, see Selwyn, 2020; Selwyn et al., 2021) and made available to students and teachers through composite grade point averages. These data were then in turn used in the context of the school career in two ways: first, these data here provide the basis for reflection for the respective students, and second, teachers use the same data to counsel students regarding their learning development and further school career (ibid.). In this context, the authors appeal to discuss the use of data and the increasing datafication in schools controversially and also not to accept everything as “taken for granted” regarding the use of data (ibid.).

4.2.3 (Data-Based) Decision-Making in School

Crosswise, the pedagogical data practices described before, practices of decision-making emerged in different contexts and mostly also in connection with other pedagogical (data) practices. Data-driven decision-making in school describes “[…] the systematic use of information by administrators and teachers to improve instruction and other school practices. […] Data-driven decision-making is said to enable teachers, administrators, and other school personnel to address student learning based on documented evidence rather than merely practitioner intuition” (Kennedy and Datnow, 2011, p. 1247; see also Datnow, 2011; Mandinach, 2012; Schildkamp and Kuiper 2010; Datnow and Park, 2018). The overarching goal is to improve school in all its processes by using various kinds of data (Mandinach, 2012; Espin et al., 2017), and so “to bring evidence to light that will help educators think about student achievement in a new way” (Datnow and Park, 2018, p. 144). But following Datnow and Park (2018), our interest is to look at the practices that arise as a result. So “although the term data-driven decision-making is commonly used in the field, […] we will refer to the practice as data-informed decision-making to signal this important shift in thinking about data use” (ibid., p. 135). Data practices of decision-making, for example, were shown to be directly related to diverse practices: from practices of assessment to practices of teaching and learning (see section 4.2.1; Bowers, 2009, p. 609f). In the context of decision-making regarding performance assessment and prediction (Selwyn, 2020; see also Park et al., 2017), Selwyn and others (2021) also describe the use of attendance data by teachers to identify “recalcitrant students” or “troublesome students” (Selwyn et al., 2021, p. 7) and to make decisions about possible meaningful interventions based on these data (in the context of high-stakes decision-making see Vanlommel and Schildkamp, 2018). “Being able to look at something and say, right in the last week this [student] has been later and later and later to their classes. So that’s an issue […] looking for patterns” (ibid. p. 7). Data take the place of pedagogical judgments and serve as seemingly objective reference points that are used to legitimize decisions.

Data practices of decision-making are also closely related to (data) practices of teaching and learning (see before). Here too, data are used to inform lessons’ design and adaptation decisions (e.g., Petersen et al., 2017; Dam et al., 2020; Mertala, 2020). For example, Dam and others (2020) describe teachers making decisions concerning changes in their teaching based on the collected data and their comparison as part of the “plan-do-check-act” cycle. Considering this or using the previously explained example of data-based grouping of students (Mendiburo et al., 2013; Prenger and Schildkamp, 2018), it is possible to see that there’s a shift in responsibility; the algorithm itself assigns students to small groups according to their prior knowledge. The decision of the group constellation, which teachers previously made on the basis of various aspects (e.g., prior knowledge and class dynamics), was completely taken away from them here. This results in a shift of actors and responsibilities: Until now, teachers were mainly responsible for gaining insights from data. But instead of teachers, more and more algorithms (resp. data) as an unquestioned authority (cf. Park et al., 2017; Mertala, 2020) decide which students need which learning or support and who works with whom. It is important to keep in mind that the underlying parameters of this decision as well as learning, understandings, and values are not disclosed in the algorithm, and thus, a kind of “blackboxing” (Selwyn, 2019, p. 12) of decision-making processes occurs, which also undermines the expertise of teachers (ibid.). Not only this practice raises questions of responsibility and accountability of teachers and data. And the previous practices of data-based decision-making were by no means as evidence-based as mentioned but also in a position to continue writing (pre-)judgments. As Datnow and Park (2018), p. 135 highlighted, “In a study of data use, Belgium et al. (2017) found that ‘teachers seek for data that confirm what they believe to be true and avoid data that question their beliefs’” (p. 81). Data can be used to validate the existing understandings of students’ learning profiles. Ultimately, the intention of the production and use of data by pedagogical school staff also always represents a decision, whereby this has generally developed a new practice in schools in recent years.

4.2.4 Cooperation and Collaboration by “Doing Data”

The practice of collaboration is slightly different from data practices found so far. It is not so much driven by data but arises out of the engagement with data. So, some of the articles report on school data teams³ that are given the specific task of processing data (Mandinach, 2012; Schildkamp et al., 2015; Ebbeler et al., 2016). Data and data analysis as well as data processing encourage meetings and lead to communication about data or with data as a basis as organizational practice (e.g., Kennedy and Datnow, 2011; Park et al., 2017). Based on data collection and (collaborative) interpretation, organizational changes such as data discussion sessions established themselves, which help in “examining data from benchmark or quarterly assessments, rewriting test items, and ultimately, joint instructional planning” (Datnow, 2011, p. 153). Such meetings provide an opportunity for teachers to focus on student growth using data to better assess what they believe their students are capable of (Datnow and Park, 2018). According to Hubers et al. (2018), such data team interventions have two goals: “The first goal is the professional development of the data team members. To accomplish this, data team members need to collaborate and share their knowledge with each other about data use in general and about the specific educational problem they are studying. The second goal is school improvement by solving the educational problem and increasing colleagues’ level of data use” (p. 2). However, it is also reported in this context that teacher collaborations on data are also enforced, for example, by requirements of the school administration. “(T)eachers felt these prompts were “busywork,” and not a productive use of their time” (Lockton et al., 2019, p. 10). At this point, tension becomes apparent as teacher collaboration meetings oscillate between being perceived as a place for genuine inquiry into student learning and being “administratively regulated around compliance demands related to data use” (Datnow and Park, 2018, p. 153; also Datnow et al., 2013).

Another form of cooperation triggered by the use of data can be seen in the cooperation between teachers and parents (e.g., Kennedy and Datnow, 2011; Schildkamp and Kuiper, 2010; Sun et al., 2016; Schildkamp et al., 2017). Such cooperation is, for example, evident in both regular meetings to jointly discuss data and derive implications, and the sharing of assessment data and progress reports (cf. Sun et al., 2016, p. 18). Especially for the latter group of parents and the cooperation between school and parents, data-based decisions by teachers are certainly an element that would be of interest from the perspective of both support and control. There were also reports of cooperation by further stakeholders or the inspectorate of education (cf. Schildkamp and Kuiper, 2010, p. 484).

Because of the increasing datafication and digitization of school, other forms of cooperation are coming into focus, such as the cooperation between teachers and software developers (e.g., Resende, 2018), especially when considering the development of data infrastructures for schools. Data could be used directly for the design process of the instrument if teachers were involved in the process of creating it. However, such data practices of cooperation have not yet been shown in the studies. Zupanc et al. (2009) also argue for closer cooperation between research, policy, and practice as a stimulus for improving educational quality. They state that the “ […] power of the data are (sic!) not so much in the numbers but in the discussions related to those numbers” (p. 306). Data then enable both the diagnosis of strengths and weaknesses in the learning concepts for teaching improvement (see before) and a shared problem analysis/discussion to stimulate educational quality improvement (see Kennedy and Datnow, 2011; Selwyn and Pangrazio, 2018).

5 Conclusion and Implications

In the last section, we already mentioned particularities and gaps in the literature, which we now want to systematically bring together. To this end, we first highlight the perspectives taken in the literature before identifying research gaps in the form of three blind spots and discussing possible developments in the field of datafication in schools that became apparent during the review. The article concludes with a brief outlook on future challenges for research on data and datafication in schools and related data practices.

5.1 Perspectives in Discourses About Data (Practices)

All articles oscillate between the challenges of datafied education and thus between the positive aspects of individualization and adaptivity of instructional design as well as the measurement of education and the construction of data regimes in schools (see, e.g., Selwyn, 2020). The analysis of theoretical and empirical articles on data, datafication, and data practices in schools showed furthermore that a large amount of studies considers this topic from the perspective of learning analytics or educational data mining. However, perspectives of practices are rather rarely explicitly mentioned in the articles examined; many studies deal with data practices but do not name them as data practices or do not place the individual practices in the larger discourse, but rather describe individual practices. Nevertheless, two perspectives could be identified in the literature:

From data to data practices …

We followed in our theoretical background the reading that data practices are constituent elements of social life and are interested in how actors relate to and begin to work with data. This understanding of practices related socio-materiality means to conceptualize technologies in their constitutive participation in practice (cf. Röhl, 2013, p. 2). It was therefore relevant for us to look at “doing data” (ibid.) in schools. In our study, just the more recent publications have shared this perspective and take a look at data and digitization in school practices together (e.g., Park et al., 2017; Selwyn, 2020). In this context, data practices receive attention in the current discourse against the backdrop of their performative character; for example, (Ruppert et al., 2013) emphasize that it is precisely the performativity of devices and data, their composition, connections, and relationships to other actors and “their multiple arrangements and mobilizations, and, of course, their instabilities, durabilities, and how they sometimes become disaggregated too” (2013, pp. 31-32) that should be the focus of (critical) attention and thus the question of what data practices do and what their effects are. Data practices, as Decuypere (2021) points out in this context, namely, redefine social life [and schools] as we understand it, act in it, and shape it precisely because of this performativity (cf. p. 69). The focus on “doing data” (Selwyn, 2020, p. 3)—especially in a school context—thus expands the perspective of actions with data and their consequences to the different ways in which data are “brought to life” (ibid.) and how actors refer to data in this regard and begin to work with data in practice (ibid.). But these also reciprocally influence or even enable ways of thinking and acting on the part of the actors (Selwyn and Pangrazio, 2018). Thus, data practices are no longer understood as acting with neutral tools “that merely ‘discover,’ ‘objectify,’ or ‘measure’ educational practices and activities” (Decuypere, 2021, p. 68).

The majority of the articles however understand data practices in the general sense of the word (ibid.), according to which they encompass the actions, performances, and resulting consequences that evoke an introduction of data-producing technologies in everyday educational situations (ibid., see also Smith, 2018). Here, on the one hand, we find publications that highlight the opportunities and potentials of using learning analytics and educational data mining in the sense of data positivism (e.g., Zupanc et al., 2009; Mendiburo et al., 2013; Boulden, 2015; Hershkovitz, 2015; Roll and Winne, 2015; Doko and Bexheti, 2018; Brun et al., 2019). On the other hand, critical articles can also be found that increasingly point to the dangers of increasing datafication and emphasize the need for reflection on the use of digital tools or learning analytics and educational data mining applications (Selwyn et al., 2015; Thompson and Cook, 2017; Resende, 2018; Ebner, 2019; Macgilchrist, 2019; Selwyn, 2019; Mertala, 2020; Selwyn et al., 2021). It is noticeable here that the articles of the positive position mostly negotiate a specific tool or the development of algorithms in terms of content and in this context also highlight (especially) their opportunities, whereas the critical articles mostly take a more general view of learning analytics and the use of data, which may be a possible explanation for the respective foci or perspectives. Overall, only a few publications can be identified that present both positive and negative aspects in equal terms, thus taking a “neutral” position in the discourse (e.g., Papamitsiou and Economides, 2014).

… and those involved in them

Closely related to the perspectives taken on data and data practices are the addressing of target groups or the consideration of actors in the context of studies. With regard to this, it becomes clear that most studies address pedagogical staff like teachers and school leaders. Therefore, it seems almost obvious that Bowers (2009), for example, describes teachers as people who generate data (in the form of grades). For example, Selwyn and others first whiten that engagement with data in schools and responsibility for leading and conducting data work usually rests only with individual dedicated teachers or smaller groups of people (Selwyn et al., 2015; Selwyn and Pangrazio, 2018; Selwyn, 2020; Selwyn et al., 2021). A systematic or all-encompassing engagement with data in teachers’ daily professional lives could not be found in any of the included sources. Rather, many sources focused on one area of teachers’ tasks or activities. For example, it was found that teachers who use data do so primarily at the classroom level to plan their lessons and monitor student progress (see above). In this context, data are also used, for example, to identify learners for whom interventions are needed and thus guide the teaching–learning process (e.g., Starkey and Eppel, 2019). Other publications however have examined how teachers collect and make sense of data to inform their judgments in the context of data-driven decision-making (e.g., Vanlommel and Schildkamp, 2019; Prenger and Schildkamp, 2018). Strikingly, many studies report the common problem that teachers are unable to interpret data or find it difficult to do so (e.g., Dam et al., 2020; Mertala, 2020; Starkey and Eppel, 2019; Ebbeler et al., 2016; Omoso, 2012). There are similar findings with regard to data from comparative tests such as PISA or ICILS, which teachers are unable to interpret sufficiently (for German teachers, see Koch, 2011). Emphasis is often placed here on the need for teachers to be (pedagogical) data literate (see Mandinach, 2012; Mandinach and Gummer, 2016; Ebbeler et al., 2016; Datnow et al., 2012) in order to consequently draw on multiple data sources continuously, effectively, and ethically and to use the data collected for improvement in accordance with teachers’ professional roles and responsibilities (cf. Vanlommel and Schildkamp, 2019, p.3). Thus, the data-related professionalization of teachers is increasingly coming into focus (Wayman and Jimerson, 2014; Mandinach and Gummer, 2016; Ebbeler et al., 2016). However, it must be mentioned in this context that “a related problem is that there is no clear agreement on what data literacy is or how to operationalize the construct (…)” (Mandinach, 2012, p. 80), even though data literacy seems to be necessary to enable educators and teachers to engage with data and make datafication productive⁴ (Gapski, 2015; Zorn, 2017). This requires the development of concepts for data education as the critical examination of data and its implications in the shaping of schools and the related professionalization not only of teachers but also of students (e.g., Zupanc et al., 2009; Selwyn, 2019).

Beyond teachers as actors in data practice, individual studies also offer implications for other target groups: On the one hand, (predominantly critical) explanations also address educational policy as an actor for whom opportunities while also bearing responsibility for critical reflection (e.g., Zupanc et al., 2009; Starkey and Eppel, 2019). On the other hand, technologies and software developers are also mentioned in single articles. For example, Mandinach (2012) points out that educators are confronted with a wide range of data that exceeds human capacity (cf. p. 73), and therefore, ways must be found to cope with these volumes of data. Here, technological solutions come to the fore to support data-driven practices that help pedagogues “collect, analyze, and report data in meaningful ways” (ibid., p. 76). Another publication is dedicated to the role of software developers and thus to the perspective of software affordances (Macgilchrist, 2019), although implications for software developers can also be found implicitly in other publications (e.g., Brun et al., 2019). Similarly, only one other study focused on librarians as another non-educational group of school actors (Boulden, 2015). Other school stakeholders are not mentioned.

5.2 Current Research Gaps and Blind Spots

Summing up, there are different blind spots and research gaps in the articles found. One blind spot are the targeting groups: For example, mostly all articles address teachers and principals, some students, or in cooperation practices parents—there are less articles addressing other pedagogical staff in schools like school social workers or media educators. Furthermore, only one article addresses decidedly the role of software developers (Mcgilchrist, 2019), whereas media or technologies as a constitutive participant are either only mentioned (e.g., Mandinach, 2012) or do not receive any attention per se.

Thus, it is not considered how exactly technologies are interwoven with pedagogical practices in schools and therefore enable but also restrict action from a praxeological point of view. Critical reflection and examination of technologies as actors of practices and the assumptions prefigured by technologies (e.g., Kitchin and Lauriault, 2014; Lachney et al., 2016) are also missing in the discussion so far. Especially in view of current debates about artificial intelligence and self-learning algorithms, however, it seems increasingly necessary to also look at software itself, what becomes visible through it, but also what is or cannot be depicted by software, and how software and therefore data change in order to generate implications for school actors and the use of technologies.

Another blind spot concerns methodological perspectives on the datafication of schools: “Methodologically, all this implies that researchers who are investigating data practices should refrain from making sharp a priori demarcations regarding what would be part of a data practice and what not” (Decuypere, 2021, p. 79). Studies also always shed light on data practices of the researchers and frame them accordingly (see Sellar, 2015 and the interconnectedness of study and research objects). However, data practices of researchers themselves have hardly been addressed or reflected upon, so further research and discussion is needed.

5.3 Limitations and Future Challenges

Finally, with regard to the research findings in the context of this review, it is important to discuss that by focusing on pedagogical practices in schools, a large number of publications that take a different perspective have been excluded. For example, studies on school management at a higher level, at which data per se play an important role, were not considered as pedagogical practices are not inherent here in our understanding. Although the second research phase and the addition of studies through references from experts in the field led to the inclusion of further publications, the data corpus is still relatively small. Therefore, no claim to completeness can be made at this point.

Nonetheless, the literature review highlighted challenges in the context of pedagogical (data) practices in schools that offer implications for future research and also for schools: The production and use of digital data are increasingly transforming pedagogical practices in schools; nonetheless, it can be stated that schools are not “improved” or “more efficient” only through the use of digital data (Selwyn et al., 2015). Thus, the increasing amount of digital data does not equate to a comparable increase in knowledge. Instead, it seems that digital data in these schools—as elsewhere in society—are often used in ways that make power increasingly “invisible” and “taken for granted.” Or, to say it with Lewis and Holloway (2018) (p. 14), “This in turn reshapes the teaching professional, in the sense that teachers are valued most for openly professing a data-responsive disposition and for their ability to embody these data-informed renderings of self, over and above other more educative and pedagogical practices.” This makes it necessary to look critically at datafication not just in schools (Roberts-Holmes and Bradbury, 2016; Selwyn, 2019) and to broaden perspectives in theoretical and empirical examination of this issue (e.g., with regard to the addressed target groups). In the future, it will be important not only to consider the interweaving and mutual interaction of the discourses around datafication, digitization, learnification, and accountability but also to address questions about data competencies of teachers as well as students, and to focus on a critical, reflexive approach to data practices on the part of researchers themselves. This interconnectedness and the still widespread data positivism need to be addressed and critically reflected upon in the future.

Author Contributions

UK and MS-R examined the articles to address research questions and wrote the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.

Funding

This article had received funding from the Federal Ministry of Education and Research Germany (01JD1903A) and the University of Kaiserslautern.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to thank Michael Becker, Christian Layes and Lorenz Wagner for their support in the literature research and Anna Hartenstein for critical proofreading. Special thanks are due to the reviewers of this article for their helpful advice and comments.

Footnotes

¹Since the publication date of this article was delayed, a second search was conducted in 2021 to ensure that as many relevant articles as possible were found.

²For details on the discourse around so-called “bubble kids” and educational triage practices, see Schildkamp and Kuiper, 2010 and Datnow and Park, 2018.

³Data teams in this context usually consist of several people, such as principals, teachers, and also quality managers, who meet at regular intervals to collaborate on data-based work on school related problems (Hubers et al., 2018; Mandinach, 2012; Schildkamp et al., 2015; Ebbeler et al., 2016; Park et al., 2017).

⁴Above all, the relation of data education to other concepts in the trans-/interdisciplinary field needs to be clarified, f.e., to datafication as a process of intertwining social and digital processes (Breiter and Hepp, 2018; Jarke and Breiter, 2019); data competences or sovereignty (Kirschsieper, 2016; Dander, 2018b); data literacy (Kaiser et al., 2018) or personal data literacies (Pangrazio and Selwyn, 2018); statistical literacy (Ben-Ziv et al., 2004); critical information and media literacy (Schiefner-Rohs, 2012); and discussions about the quantified self (Swan, 2013; Allert et al., 2017).

References

Abrams, L., Varier, D., and Jackson, L. (2016). Unpacking Instructional Alignment: The Influence of Teachers’ Use of Assessment Data on Instruction. Perspect. Edu. 34 (4), 15–28. doi:10.18820/2519593X/pie.v34i4.2