An equitable and scalable approach to track fidelity of implementation in partnership with teachers

Social and emotional learning (SEL) is a key focus of early childhood education. A significant body of research demonstrates the critical impact of fidelity of SEL curriculum and intervention implementation on child outcomes; however, few widely-used curricula regularly assess classroom-level implementation fidelity outside of the context of research or correlate fidelity with key areas of children’s development of these skills. Fidelity measurement often focuses on easy-to-quantify variables such as classroom environment or lesson plan adherence, and is a periodic snapshot used as a moderator or co-variate when assessing child outcomes, rather than being intentionally leveraged as a systematic, ongoing process to evaluate and support implementation. In this paper, we present a novel approach to capturing fidelity data as a core component of professional development. We outline our findings from a pilot of our approach using short, teacher-recorded videos submitted across the school year as a vehicle for capturing and sharing real-time data related to professional learning, implementation, and curricula impact, as well as a framework for building equitable partnerships with teachers. Results from the initial pilot of this approach in several hundred classrooms across the US demonstrate feasibility and utility and suggest that teacher-recorded videos can offer a scalable means to collect continuous samples of fidelity data, providing a richer view of professional learning, while simultaneously creating the opportunity to provide ongoing feedback and engage teachers in partnership in reflecting on practice and its impact on children’s development. We developed and piloted an approach where teachers record and upload videos of teaching practices and children engaging with their peers in specific classroom activities via a mobile application. Each video submission has a focal activity and associated set of indicators which are shared with teachers in advance to create an equitable feedback system in which both curriculum staff and teachers engage in reflecting on children’s interactions and the application of the curricular approach in their classroom. Videos are viewed and coded on these sets of indicators by both the teachers and curricular coaches who provide targeted feedback in an interactive exchange on a dashboard accessible by teachers and their curricular coach.

Introduction

Social and emotional learning is a critical component of early childhood education and a target of early childhood interventions and curricula. This key area of development is emphasized in the early childhood years because the social and emotional skills learned early in life serve as a foundation for future social interactions, emotion regulation, and are linked to improved life outcomes in school and beyond (Moffitt et al., 2011; Jones et al., 2015; Taylor et al., 2017). A significant body of research shows that when a curriculum is implemented with high fidelity, there is a positive impact on children’s academic and social–emotional outcomes (Durlak and DuPre, 2008; Hamre et al., 2010; Durlak et al., 2011; Sklad et al., 2012; Quinn and Kim, 2017).

While extant research shows a strong relationship between fidelity and child outcomes, the practice of measuring fidelity to a curriculum remains highly variable, with practices ranging from ratings by program experts via in-classroom observation, videotaping, interviews, activity checklists and/or record reviews (Mowbray et al., 2003; Bickman et al., 2009). There is a need for continued work in the field to better operationalize fidelity in a way that provides teachers ongoing support, an equitable experience, and enables high-level implementation (e.g., Pianta, 2005; Landry et al., 2006; Pianta et al., 2008).

Measuring fidelity of implementation is a vital factor in examining the impact of interventions and curricula (de Leeuw et al., 2020), but measuring implementation in a supportive, cost-effective and scalable way has not always been feasible (Bickman et al., 2009). Traditional observation-based implementation fidelity data collection requires observers to be highly trained in the curriculum and fidelity measure, deemed reliable on the measure, and complete in-person classroom observations or video data collection multiple times across a school year. This approach requires training and travel time, both of which involve additional costs. While these observation-based methods of measuring fidelity are the “gold standard,” executing them in person is time and labor-intensive, resulting in higher costs and challenges to scale (Huntley, 2009; Barton et al., 2017). In addition, teachers can be wary of outside observers in their classrooms, assuming evaluation or fearing punitive outcomes as a result of what is (or is not) observed (Shernoff et al., 2017). These approaches can often leave teachers in the dark in terms of what is being observed, how they are doing in implementing the intervention or curriculum and creates an inequitable relationship with teachers.

Adding an additional layer of challenge, fidelity measurement can be more complicated in interventions and curricula where practice may involve responsive differentiation of instruction and multiple activities. A window of opportunity to address these challenges coincides with the fact that educators implementing new curricula and interventions need professional development to support implementation of the curriculum to fidelity (Pianta et al., 2008).

In response to these challenges, Tools of the Mind developed TREE (Teachers Reaching Educational Excellence), a novel, scalable approach that re-conceptualizes fidelity measurement as part of on-going professional development. In this paper, we describe our approach and our experience piloting TREE in classrooms learning the Tools of the Mind curriculum during the 2021–2022 school year to illustrate the feasibility and promise of using teacher-recorded video data to measure fidelity of implementation and provide individualized professional development to support continuous learning and mastery. The use of video to provide a “window into practice” provides teachers the opportunity to reflect on their practice in a way that is not possible while in the midst of teaching (Clarke and Hollingsworth, 2002; Borko et al., 2008; Zhang et al., 2011; Marsh and Mitchell, 2014), and a cost-effective and wider-reaching solution for curricular coaches to work in tandem with teachers to review, reflect and support what is happening in the classroom. It is also a more cost-effective approach in comparison to higher cost repeated in-person coaching visits (Dede et al., 2009; Barton et al., 2017).

Fidelity of implementation has been defined in a range of ways, including adherence to an intervention or program as designed by the developers, “strict adherence” to methods and theory, “completeness” of implementation, or the quality of program delivery (Dusenbury et al., 2003; Carroll et al., 2007). A definition that emphasizes adherence to methods and theory is most appropriate for an intervention or curriculum such as Tools of the Mind (Tools), that views teachers as central in its theory of change and highlights the role of educators and their professional learning as a critical factor in fidelity of implementation. Defining fidelity of implementation as the skillful application of methods and theory (Dusenbury et al., 2003) opens the gateway to a continuous approach to measuring fidelity of implementation. A continuous approach is well-designed to capture the developmental trajectory of professional learning as teachers are applying new knowledge, teaching practices and strategies in new activities.

Early childhood educators implementing a new intervention or curriculum that requires adherence to methods and theory need to develop a solid understanding of and practice applying the methods and theory as well as an understanding of the key components (de Leeuw et al., 2020). Professional development and support in implementing the intervention or curriculum to fidelity would be a natural component of interventions and curricula that build teacher understanding to apply an approach and methods. Early childhood interventions and curricula rely on teachers as the key factor in the equation to impact child outcomes; fidelity measurement in turn, needs to capture the developmental trajectory of professional learning as teachers are applying new knowledge, teaching practices and strategies in new activities in a scalable, reliable and feasible way.

This paper offers a case study of an approach that roots fidelity measurement in ongoing teacher professional development to better support teachers as they learn, adopt, and utilize a curriculum. This approach engages teachers in the opportunity to reflect on practice and receive individualized feedback at the same time as a stream of real-time data capturing implementation, teacher learning, and a developmental trajectory of teachers’ fidelity of implementation is collected.

There are five overarching ideas that may prove useful to others in the field that come out of the pilot of TREE discussed in this paper: conceptualizing fidelity of implementation as a developmental journey; integrating fidelity measurement and responsive support of teacher learning; teachers benefit when fidelity of implementation measurement and tracking is an equitable partnership; and teacher-recorded video data provides a feasible and cost-effective approach that offers significant long-term benefits for learning and research.

This approach can offer a transformative model of fidelity measurement with application to a variety of SEL interventions and curricula that have teachers at the center of their theory of change, and value creating an equitable relationship with teachers, while offering a cost-effective and scalable approach to fidelity measurement.

Pedagogical framework

Tools of the Mind curriculum

While TREE was developed in the context of a specific curriculum (Tools of the Mind), we believe that the principles and methodology are broadly applicable to the field and can be used as a case study for classroom-based curricular interventions that are working to improve child outcomes in early childhood education. Tools of the Mind (Tools) is a comprehensive, research-based, early childhood curriculum rooted in Vygotskian theory and designed to embed self-regulation and social–emotional development in comprehensive PreK & K curricula, that has been identified as a CASEL-SELect program [Collaborative for Academic, Social, and Emotional Learning (CASEL), 2022]. From its very start, Tools was developed in partnership with teachers.

Over the last 30 years Tools has grown from a university-sponsored research project to become an independent nonprofit working with school and community-based partners across the country. Tools mission is to empower teachers with the tools they need to build inclusive classroom communities, leverage playful learning and individualize instruction to enable every child to reach their full potential. Tools has a decades-long track record of using research to continuously improve its program and approach to measuring fidelity of implementation, learning from participation in multiple research studies (e.g., Diamond et al., 2007, 2019; Barnett et al., 2008; Blair and Raver, 2014).

Tools’ underlying theory and understanding of development extends to our design of teacher professional development. Over the course of learning Tools across their first year, teachers internalize concepts, theory and neuroscience research that applies to practice; ‘learning by doing’ by taking concepts taught in core workshops, and applying them in their classroom, reflecting on their impact on children through recording and sharing video on TREE. As teachers progress along their unique developmental trajectories, they are incorporating new approaches to teaching, implementing curricular strategies, and learning through and alongside the children in their classroom. We believe this continuous process of learning provides a unique opportunity to impact teacher and child development through providing responsive, customized scaffolding.

TREE background and motivation

The TREE approach to individualized teacher professional development and measuring fidelity came out of years of developing and using different approaches of capturing fidelity of implementation of the Tools program. Over the past two decades, multiple variations of fidelity measurement tools were used, including one that focused on all of the steps and their sequence for each Tools activity. The lesson from these earlier iterations revealed that the definition of fidelity of implementation, and how it is communicated to both teachers and researchers has an impact on implementation and child outcomes. This has led us to simplify our curriculum manuals and completely redesign our professional development, engaging in iterative research on teacher and child impacts through our data share partnerships with programs implementing Tools. It also led to a clearer conceptualization of fidelity of implementation and a new approach to capturing and measuring it through focusing on a smaller set of indicators that evolve over time. This allows Tools teachers to focus on the teaching practices and child interactions and actions that matter most–the “critical components” that are vital to ensuring the program is being implemented (Century et al., 2010; Stains and Vickrey, 2017). It also provides teachers transparency and an implementation road map with clear markers to focus on over time, and individualized support to enable teachers to gain a deeper understanding of the Tools approach, and how to apply it in their classrooms. In concert, identifying core components of the program and providing them clearly to teachers, strengthens fidelity of implementation (de Leeuw et al., 2020).

Prior to the creation of TREE, Tools provided coaching and captured fidelity of implementation through in-person technical assistance visits to schools. These visits would take place at one or two points during the year. The logistics of in-person visits meant that Tools’ curricular coaches rarely were able to visit every classroom in a school to observe key teaching practices and activities in action and were unable to individually engage in 1:1 coaching with every teacher. In designing TREE, a key motivation was to find an alternative approach that would allow for more immediate and consistent feedback to every teacher, while also capturing and measuring fidelity of implementation over time.

As a curriculum that has from its origin been co-constructed with teachers, equity in our partnership with teachers is a core value. We believe that the measurement and tracking of fidelity of implementation can and should be inclusive of teachers as collaborators. Historically in the majority of our research experiences, teachers were blind to what was being looked for in in-person fidelity measurement observations and video-recordings. They did not have the opportunity to learn about the impact on children until the end of year, or sometimes several years later, and were often blind to how their fidelity of implementation was scored. Engaging teachers in actively observing the impact of their teaching practices on children and connecting how they implement the curriculum to what they observe in children’s ongoing development has tremendous value for them.

Building on the design of Tools’ curriculum and professional development, we approached the creation of our fidelity instrument through a developmental lens. Rather than starting with a notion of ‘perfect implementation’ of a teacher with significant experience, we began by creating a developmental trajectory of Tools teachers’ growing mastery applying Tools approach, and implementation of the curriculum to create fidelity indicators over time based on our prior data from in-person technical assistance visits. We also reviewed previously used fidelity assessment measures, distilling the ‘key steps’ in Tools activities, as well as creating indicators to capture teachers’ responsive increase of challenge level and modification of teaching practices to support children as they develop across the year.

TREE learning environment and pilot learning objectives

Developing an integrated model of professional development and fidelity measurement

Tools’ model of professional development consists of a foundation of live interactive workshops spread across the year. In workshops, teachers learn Tools’ theory, how to implement Tools activities, and strengthen their knowledge of child development, while developing new teaching practices and strategies to scaffold and individualize instruction and build inclusive classroom cultures.

These workshops are paired with small-group Tools-facilitated Professional Learning Community (PLC) live virtual sessions focused on team building and co-construction between teachers. These sessions provide teachers with a cohort of peers to engage in discussion based on the groups’ collective needs and challenges as they progress across the year.

The last component is our individualized professional development on TREE. TREE provides teachers with transparent information about the anticipated trajectory of implementation with a roadmap with clear benchmarks for implementation over time. The TREE model integrates professional learning with an approach to capturing the trajectory of fidelity of implementation in early childhood classrooms. Conceptualized over a 2-year period, segmented into the 1st and 2nd years of professional learning, with a subsequent 3rd year option to become certified as masterful Endorsed Tools teachers, this model builds a bridge between theory and practice. TREE engages teachers in closely observing and capturing videos of their teaching practices, children’s actions and interactions, then viewing the video to reflect on practice and the indicators they are focused on in a given month. TREE creates an environment in which the fidelity of implementation in the classroom is captured while simultaneously supporting the individual trajectory of implementation development for each teacher.

Pilot study

In the pilot study of TREE we had the goal to assess the feasibility and usability of teacher-recorded video and identify if it is a cost-effective approach to capturing fidelity of implementation in an embedded system of individualized professional development.

Our primary learning objectives include:

• Is teacher-recorded video feasible and useful for fidelity assessment?

• Will teachers find the TREE approach supportive?

• Will our curricular coaches find TREE feasible and useful?

• Will this approach be cost-effective and scalable?

Participants

A total of 301 PreK and Kindergarten teachers from 65 different school districts across the United States took part in the initial pilot of TREE during the 2021–2022 school year. Teachers had between 0 and 34 years of teaching experience, with a median of 7 years of experience in the classroom. Participating school districts varied in both socioeconomic and racial/ethnic diversity, as measured by publicly available data of the percentage of students eligible for free or reduced-price lunch and Ethnic Diversity Index data. School districts in the pilot had a median of 43% (IQR = 33%) of students eligible for free or reduced-price lunch, and a median ethnic diversity index score of 0.59 (IQR = 0.28) as shown in Table 1.

TABLE 1

Table 1. Participant demographics.

Pedagogical framework of TREE

Anatomy of TREE cycles

In TREE, there are series of video cycles in each year teachers are learning and mastering Tools’ approach, with select focus areas that capture children’s engagement in learning activities, interactions, and teachers’ use of strategies in Tools activities designed to support self-regulation and social skill development as well as academic skills. The teaching practices and strategies taught in professional development become distilled into select actionable and measurable indicators in each video cycle. The indicators in a given activity are selected because they capture child behavioral impacts of fidelity of implementation of teaching practices or capture the teaching practices in action (see Figure 1). These indicators are chosen because they are generalizable teaching and learning practices that are applied in multiple Tools activities across the day, so teacher and/or child mastery in one activity extends across the day and correlates with fidelity of implementation.

FIGURE 1

Figure 1. Tools TREE cycle.

Cycles are conceptualized as an equitable feedback system between the teacher, their curricular coach, and their peer PLC group as displayed in Figure 2. The format of each cycle consists of two video submissions, punctuated by a narrative feedback loop with the curricular coach tagged to key moments in the video, and a facilitated PLC meeting with their cohort.

FIGURE 2

Figure 2. Teacher TREE growth visualization.

Recording and submitting video

For each focus area, teachers record a short (≤8 min) video of children’s actions and interactions in an activity as well as teacher interactions with children. Teachers are reminded frequently that the aim is not perfection, but instead to capture current development, as an opportunity for reflection and learning, to propel teachers’ and children’s development. Prior to submitting their video, the teacher reviews the indicators associated with the focus area and identifies those that are observed in their video. This encourages the teacher to reflect on their own practice and children’s interactions using the same measurement tool that their curricular coach will use. This performs a central function in the equitable feedback system by providing a core set of achievable goals that support teachers’ intrinsic motivation, ownership, developing understanding, reflective practice, and growth. Over time, TREE provides teachers a visualization of their integration of Tools approach and implementation of the curriculum in their classroom, and continuous growth across the year (see Figure 3), and they can look back in year 2 at where their children were, and what they were doing at a similar time in the previous year.

FIGURE 3

Figure 3. Tools Indicator example.

Feedback cycle

Following the submission of a video and self-identification of indicators, teachers receive feedback from their coach through an interactive interface that allows for the coach to tie written comments to time-stamped sections of the teacher’s video submission. The coach may ask questions and comment on observed areas of focus to identify strengths and make suggestions about next potential steps based on teachers’ and children’s current levels of development. Teachers provide feedback to coaches about what is most applicable and helpful, as well as ask and respond to questions. The platform allows teachers to reflect on the coaches’ feedback, rewatch key areas of their video, and decide what strategies they want to focus on to strengthen their implementation, and children’s outcomes.

Professional learning community (PLC) meetings and shared learning

After receiving feedback from their coach, the teachers participate in facilitated PLC cohorts where they can bring questions and discuss topics relevant to the current area of focus to deepen their understanding of core teaching practices and the curriculum alongside their dedicated coach and colleagues. This time is used to deepen understanding of specific concepts, brainstorm strategies for common classroom scenarios, provide support, and build camaraderie.

Visualization of growth

Following their PLC meeting, teachers record and submit a second video of the same focus area and indicators, applying their strengthening understanding of the theory and child impact goals from their reflective practice, coaching and PLC discussions. Teachers view their video, identify the indicators they observe, write questions or notes to their coach, and their curricular coach provides feedback and captures the teacher’s current fidelity of implementation by checking the indicators observed in the video. When the coach has completed this, the teacher’s TREE dashboard animates to highlight growth in the form of a tree representing each video focus. Each tree starts from a seed that germinates, sprouts, and matures into a sapling, followed by a mature tree that blossoms and bears fruit. The teacher then begins the next cycle, represented by a new tree with a different focus area and indicators tied to that tree’s growth. The tree visual mediator follows the teacher throughout the year, and over the course of a two-year period of professional development, after which teachers have the opportunity to become endorsed.

The curricular coach TREE experience

Our curricular coaches have their own TREE dashboards, which enables them to see the development of fidelity of implementation across all their classrooms, as well as displaying data at a school and individual classroom level. This data is used to inform customization of PLCs and workshops to meet teachers where they are and support ongoing learning and implementation.

For the curricular coaches providing dedicated support to teachers, TREE provides a modality for capturing fidelity of implementation, interaction and feedback that bypasses many challenges associated with traditional models of technical assistance. TREE provides a glimpse into the classroom, as one Coach reflected, “at exactly the right moment,” and equips coaches with an interface that supports their workflows. Dashboard features, like the time-stamp functionality and categories of feedback, make engaging with the content easy, meaningful, and systematized. Coaches can reflect on the similarities and differences between their and teachers’ observation of indicators and apply this awareness in their coaching interaction. Coaches can rewatch a video or rewind it to better observe or hear something – something not possible in live in-person visits. Capturing fidelity of implementation involves checking a short list of indicators, the same short list of indicators across all the classrooms they are coaching in a given month, building their capacity and experience, and making fidelity of implementation capturing more accurate.

For many coaches, TREE provides, for the first time, the ability to see the impact of their scaffolding – with visibility into whether more indicators are captured in the second video of the same activity in each cycle, and what new strategies are adopted by teachers to strengthen application of Tools approach and curriculum implementation.

TREE training process for curricular coaches

Our curricular coaches are employees of Tools of the Mind, and former Tools of the Mind teachers or coaches. Initial training for coaches starting to use TREE was over the course of 3 days. This training introduced the video focuses and indicators, how to download the app and navigate the platform and engaged coaches in shared practice sessions with video. Following this, coaches had monthly 2-h professional development sessions focused on each set of indicators and how to identify them, and how to apply Tools theory to support teacher development, with a focus on identifying the next steps that teachers will be able to independently apply after coaching to support their continuous growth in implementing Tools.

This training has enabled Tools to onboard new team members, oversee and support development of reliability in fidelity measurement, and monitor and learn about the effectiveness of coaching strategies. This combination of data visibility and shared learning has been an effective way to support team growth.

Results

Participants access to technology

All participating teachers had access to high-speed internet, and 83% of participating teachers chose to install and use the TREE mobile app available for Apple iOS and Android mobile devices. The remaining 17% utilized our web browser-based alternative for uploading video.

A total of 1,140 videos were recorded and submitted by teachers during our TREE pilot in the 2021–2022 school year. Submitted videos had a median duration of approximately 7.3 min (Median Duration = 440 s; IQR = 392 s), and median file size for uploaded videos was 491 MB (IQR = 572 MB). The majority (90%) of videos submitted by teachers showed evidence of targeted classroom practices, and few teachers reported difficulty with recording videos (Table 2).

TABLE 2

Table 2. TREE video descriptive statistics.

Feasibility and utility

Overall, internal product feedback and data usage from both teachers and curricular coaches indicated that the TREE model was convenient, user-friendly, and helpful in daily teaching practice. 100% of teacher-recorded videos were correctly formatted and submitted.

Of the teachers responding to a feedback survey (n = 81), 91% of teachers in their first year of learning Tools felt supported by their experience with TREE. One teacher relayed that they “felt extremely motivated after each video feedback. The suggested ideas were always so beneficial. […] It was truly a valuable support to have while learning a new curriculum,” and another wrote “It is so worth it. The insight you get back not only improves your teaching strategies but the students’ learning.”

In terms of user experience, our curricular coaches reported that teachers had few difficulties submitting videos and using the TREE interface, one coach writing “even my teachers that were reluctant in using the mobile app thought it was easy to use,” and another said, “teachers seemed to like the ease of it.” A key concern when introducing any new technology is the amount of technical support required to ensure user success in adopting and using it. We provided users with support via email and our website during business hours, and an analysis of support request tickets shows 35 unique TREE support requests across the year, the majority of which occurred at the launch of TREE rollout as teachers learned how to upload video. All were successfully resolved. There were no long-term technology barriers that created challenges to using TREE.

The TREE platform was also highly regarded by our coaches, who valued the ability to link feedback to specific moments in the video and look back on past video and feedback histories. A coach shared that TREE “...allows me to prompt the teacher – look at minute x and this child – what do you notice?.” This pilot highlights the feasibility of partnership with teachers for the recording and sharing of video in their classroom using mobile devices for measuring fidelity and individualized professional development, as well as the feasibility for coaches who are reliably capturing fidelity of implementation and providing individualized support.

Cost-effectiveness and scalability

Staff time

In the context of measuring and supporting implementation fidelity in both research and professional development contexts, the demands of research and/or coach time are frequently a limiting factor in terms of scale. When assessing fidelity directly via in-person observation or in-person recording of video, logistical constraints such as travel time and classroom schedules impact the number of classrooms that can be observed per day. An important question for Tools was how utilizing teacher-recorded video would impact the ability to scale our reach and improve the efficiency of individualized professional development to maximize impact. In order to answer this question, Tools designed the TREE coach portal used by our team to review and respond to videos which allowed us to also internally track the time spent by team members on each video down to the second. Our findings suggest that the use of teacher-recorded video in TREE was both efficient and cost-effective compared to in-person observation. The median review time for submitted videos was 17 min (IQR = 20 min), which included reviewing the video, coding indicators present or absent, and communicating with teachers the time-stamped strengths they observed, and strategies they could apply to strengthen practice and support children’s development of target skills.

Security and storage costs

Protecting children’s privacy is fundamental to Tools of the Mind’s work with teachers, schools, and school districts. To this end, we ensure that data storage and transmission are end-to-end encrypted and utilize industry-leading cloud storage providers who conform to relevant domestic and international security standards for data security such as SOC 1/SSAE 16/ISAE 3402, SOC 2, PCI DSS Level 1, ISO 27001, and FISMA. Additionally, we use industry best practices and align with national and international standards for data security and privacy governance including FERPA, COPPA, state privacy laws, and global regulations.

Recent advances in cloud computing have dramatically reduced the infrastructure and cost associated with uploading and storing large amounts of video data such as those collected in our pilot. During the 2021–2022 school year, Tools collected more than 190 h of video associated with our pilot for a total of 872GB of video file data. At standard industry cloud storage rates, the cost to store these data in secure encrypted online storage works out to approximately $20.07 USD per month, or less than $0.22 per video per year.

Travel cost savings

In the last school year in which in-person support was provided to school districts, Tools spent an average of $329.27 per classroom-served on travel costs. Extrapolating based on these figures, Tools saved nearly $100,000 in travel related expenses by using TREE to provide 2 captures of fidelity of implementation development and monthly sessions of individualized professional development support for teachers in each classroom of the 65 programs participating in the pilot. These savings allow programs like Tools to serve more teachers, and underscore how the use of teacher-recorded video can significantly improve scalability and keep interventions and curriculum affordable to districts wanting to implement them.

Discussion

Our TREE pilot demonstrates that teacher-recorded classroom video offers a feasible, cost-effective, and scalable method for assessing fidelity while enabling targeted, responsive individualized professional development to support teacher learning and implementation. While the specific design and implementation of TREE is uniquely adapted to the Tools of the Mind curriculum, we believe that our experience and learning offers several important takeaways for the conceptualization and implementation of fidelity measurement in SEL interventions more generally.

The approach laid out in this paper adds to the field by providing a new way of thinking about and operationalizing fidelity measurement while providing a cost-effective and scalable solution that delivers on tracking curriculum implementation fidelity. Below we discuss each of the five overarching ideas covered in this paper in more detail.

Conceptualizing fidelity of implementation as a developmental journey

Rather than relying on a set of indicators that are agnostic to where teachers are in the process of learning to teach Tools with fidelity, and where children’s development is, we identified a set of “core” indicators (Century et al., 2010) across the year aligned with teacher and children’s development. We believe other interventions could benefit from reconceptualizing fidelity of implementation in this more nuanced way.

At the same time, fidelity to a curriculum that embeds an approach to teaching and learning can only be assessed by seeing the impact of teachers’ application of the approach on children at multiple developmental levels that change over time. Our experience has convinced us that integrating continuous, short, targeted fidelity snapshots with individualized professional development coaching support is a model that can both better capture fidelity of implementation than single measures and provide a way for intervention and curricula developers to support learning and implementation.

Measurement of fidelity and responsive support for teacher learning can and should be integrated

Our experience with TREE has strengthened our belief that fidelity of implementation and responsive support of teacher learning can and should be integrated. Tools, drawing on Vygotskian theory, works with teachers to empower them to observe children’s current levels of development, adjust the challenge levels of activities in response, and uses scaffolding to meet each child where they are and support them in reaching their full potential. We believe teachers deserve and will benefit from the same responsive support. Traditional professional development approaches, such as workshops, are akin to ‘whole group instruction’ and cannot individualize to meet each teacher where they are and incorporate an understanding of the children in their classrooms and their current levels of development. In addition to not being able to individualize and provide on-going support that fits teachers’ unique needs, these large-scale approaches to professional development often come with a high financial cost (Haymore-Sandholtz, 2002; Odden et al., 2002; Pianta et al., 2008). By integrating fidelity measurement with responsive and ongoing coaching support for teachers, TREE helps to minimize professional development and coaching costs while simultaneously measuring implementation fidelity for a larger group of teachers than in-person coaching allows.

Teachers benefit when fidelity of implementation measurement and tracking is an equitable partnership

With equity as a core value, through engaging teachers as partners in collecting and reflecting on data sampled multiple times across the year, we have prioritized transparency, collaboration, and inclusivity as key components of our approach.

In-person, observation-based fidelity measurement approaches limit the opportunity to engage teachers in self-reflection on implementation of a curriculum and its impact on children. Our experience has been that engaging teachers in capturing video and reflecting on the indicators observed in their video has real benefits for teachers, providing the opportunity to focus on children’s interactions and development, and reflect on practice and the impact of their teaching on children over time (e.g., Borko et al., 2008; Marsh and Mitchell, 2014). Although Tools’ commitment is to continuously collect fidelity of implementation data to measure our real-time impact and improve our support of teachers, we believe that researchers studying new interventions and curriculum would benefit from this approach. Beyond improving fidelity of implementation, including teachers in this way adds value to them as participants in research, and may make teachers more open to participating in future research studies.

Finally, research has shown the impact of teachers’ own SEL skills and wellbeing on students’ SEL (Schonert-Reichl, 2017); for this reason alone, as we develop and research the impact of interventions and curricula designed to impact children’s SEL, we should at the same time go about this in a way that promotes teacher learning and wellbeing. We believe that the process of having teachers partner in reflecting on video of their classroom to look for children’s SEL and other areas of children’s development empowers teachers, propels their learning, and contributes to their wellbeing.

The collection of teacher-recorded video in classrooms is feasible, scalable and offers significant long-term benefits for learning and research

Learning and research benefits

Video provides the opportunity for collaborative review and consensus coding, training and building inter-rater reliability as well as coaching skills to support each teacher and the children in their classroom (van der Linden et al., 2022).Video enables fast-cycle trials to evaluate the impact of professional development and individualized coaching, supporting innovation within the coaching staff, and leverages the potential to get rapid feedback from a video within a short period of time to observe if there is an improvement in teachers’ application of teaching practices and fidelity of implementation. Moreover, video data can be re-analyzed in the future as fidelity measures are revised, providing an on-going source of valuable data (e.g., Clarke and Hollingsworth, 2002; van der Linden et al., 2022).

Feasibility and scalability

The role of technology both inside and outside of the classroom has changed dramatically over the past decade. As of April 2021, 85% of US adults now own a smartphone (Horowitz and Graf, 2021). Our experience suggests that many early childhood teachers (including those new to teaching and those with decades of experience) are comfortable filming in their classrooms with smartphones. Smartphone video and audio quality is now more than sufficient for assessment of fidelity in a classroom setting, and our team encountered few barriers in being able to see and hear in the over 1,000 classroom videos recorded by teachers.

We also found that teachers were quick to learn how to use the TREE mobile app and platform. A brief instructional video for teachers coupled with a few slides presented in professional development workshops was sufficient to enable the vast majority of teachers in our pilot to download our app and successfully upload video with no additional support. Teachers’ only guidance on what to record each cycle consisted of the video focus (activity or time block) and the 5 indicators which were integrated into the TREE portal and phone app. We found these were sufficient for teachers to capture and upload video that enabled shared reflection on and assessment of fidelity of Tools implementation in classrooms over time.

Likewise, the vast majority of school districts and administrators expressed no concerns about having teachers film in classrooms for purposes of receiving individualized on-going professional development support. Administrators in TREE were also provided with continuous high-level data capturing implementation fidelity across classrooms, providing easy access to see how implementation was progressing to provide individual teachers with support as needed.

Finally, our experience suggests that switching from in-person observation to teacher-recorded video can offer significant gains in efficiency, allowing for greater reach to more teachers with the same resources with reduced overhead. Our data suggests it is feasible for a single team member to review, code, and respond to 20–25 submitted videos per workday, a four to five-fold increase over the number of teachers Tools team members could typically observe and provide feedback to during in-person visits to schools.

Acknowledgment of constraints

By design, this was a small pilot study focused on feasibility, and we cannot yet draw broad conclusions without further research. Future work will seek to assess the fidelity of implementation at a larger scale, identify the indicators correlated with school measures of child outcomes and quantify the impact of this new approach on teacher practice or child outcomes. Likewise, demonstrating the psychometric properties of our fidelity measure was outside the scope of this initial pilot, although it will be a focus of future work. Finally, because participating districts chose to participate and engage with Tools of the Mind, our results may not be representative of all teachers and school districts around the country. This was feasible in the population we described but may not be feasible in all settings.

While TREE was developed in the context of a specific curriculum that includes ongoing individualized professional development, we believe that this approach could be applicable to interventions and curricula that operate in other ways. A variety of interventions or curricula could engage teachers in capturing and uploading video and self-reflection as part of research efforts to look at impact and feasibility. Teacher-recorded video can be shared and uploaded on multiple commercially-available platforms and via multiple systems, fidelity of implementation indicators can be made transparent to teachers participating in research, and our experience suggests that such an approach can offer significant value.

Conclusion

We believe that our experience with TREE offers an important new approach to conceptualizing and operationalizing fidelity measurement, interweaving individualized support for teachers with real-time measures of program implementation fidelity via teacher-recorded video. The design of TREE as an equitable feedback system between teachers and curricular coaches empowers teachers to reflect on their own practices and children’s development and identify shared focuses in a transformative and respectful way. Capturing fidelity over time, as opposed to once or twice a year, while also providing support and uplifting teachers in their practice is an approach, we believe could be transformative for other programs and researchers, especially in SEL interventions and curricula looking to support teachers and to examine implementation fidelity and its impact.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: private. Requests to access these datasets should be directed to mspacciapoli@toolsofthemind.org.

Ethics statement

The studies involving human participants were reviewed and approved by Advarra. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

MS co-conceived, wrote, and edited manuscript, oversaw collection of survey data, and contributed to fidelity of implementation indicators. MV wrote and edited manuscript, contributed figures, and contributed to the design of TREE. OS wrote and edited manuscript, co-designed and developed TREE, conducted data analysis, co-conceived of manuscript concept, and contributed figures. JS wrote and edited manuscript and conducted data analysis. TM wrote and edited manuscript. BW-S co-conceived, wrote and edited manuscript, co-designed and developed TREE, and supervised manuscript preparation. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Barnett, W. S., Jung, K., Yarosz, D. J., Thomas, J., Hornbeck, A., Stechuk, R., et al. (2008). Educational effects of the tools of the mind curriculum: a randomized trial. Early Child. Res. Q. 23, 299–313. doi: 10.1016/j.ecresq.2008.03.001

CrossRef Full Text | Google Scholar

Barton, E. A., Whittaker, J. V., Kinzie, M. B., DeCoster, J., and Funari, E. (2017). Understanding the relationship between teachers’ use of online demonstration videos and fidelity of implementation in MyTeachingPartner-math/science. Teach. Teach. Educ. 67, 189–201. doi: 10.1016/j.tate.2017.06.011

CrossRef Full Text | Google Scholar

Bickman, L., Brown, J. L., Flay, B. R., Li, K., Pelham, W. Jr., Massetti, G., et al. (2009). Approaches to measuring implementation fidelity in school-based program evaluations. J. Res. Character Educ. 7, 75–101.

Google Scholar

Blair, C., and Raver, C. C. (2014). Closing the achievement gap through modification of neurocognitive and neuroendocrine function: results from a cluster randomized controlled trial of an innovative approach to the education of children in kindergarten. PLoS One 9:e112393. doi: 10.1371/journal.pone.0112393

PubMed Abstract | CrossRef Full Text | Google Scholar

Borko, H., Jacobs, J., Eiteljorb, E., and Pittman, M. E. (2008). Video as a tool for fostering productive discussions in mathematics professional development. Teach. Teach. Educ. 24, 417–436. doi: 10.1016/j.tate.2006.11.012

CrossRef Full Text | Google Scholar

Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., and Balain, S. (2007). A conceptual framework for implementation fidelity. Implement Sci. 2, 1–7. doi: 10.1186/1748-5908-2-40

PubMed Abstract | CrossRef Full Text | Google Scholar

Century, J., Rudnick, M., and Freeman, C. (2010). A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am. J. Eval. 31, 199–218. doi: 10.1177/1098214010366173

CrossRef Full Text | Google Scholar

Clarke, D., and Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teach. Teach. Educ. 18, 947–967. doi: 10.1016/S0742-051X(02)00053-7

CrossRef Full Text | Google Scholar

Collaborative for Academic, Social, and Emotional Learning (CASEL) (2022). Review programs. https://pg.casel.org/tools-of-the-mind/ (Accessed August 5, 2022).

Google Scholar

Dede, C., Jass Ketelhut, D., Whitehouse, P., Breit, L., and McCloskey, E. M. (2009). A research agenda for online teacher professional development. J. Teach. Educ. 60, 8–19. doi: 10.1177/0022487108327554

CrossRef Full Text | Google Scholar

de Leeuw, R. R., Boer, A. A., and Minnaert, A. E. M. G. (2020). The proof of the intervention is in the implementation; a systematic review about implementation fidelity of classroom-based interventions facilitating social participation of students with social-emotional problems or behavioural difficulties. Int. J. Educational Res. Open 1:100002. doi: 10.1016/j.ijedro.2020.100002

CrossRef Full Text | Google Scholar

Diamond, A., Barnett, S. W., Thomas, J., and Munro, S. (2007). Preschool program improves cognitive control. Science 318, 1387–1388. doi: 10.1126/science.1151148

PubMed Abstract | CrossRef Full Text | Google Scholar

Diamond, A., Lee, C., Senften, P., Lam, A., and Abbott, D. (2019). Randomized control trial of tools of the mind: marked benefits to kindergarten children and their teachers. PLoS One 14:e0222447. doi: 10.1371/journal.pone.0222447

PubMed Abstract | CrossRef Full Text | Google Scholar

Durlak, J. A., and DuPre, E. P. (2008). Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am. J. Community Psychol. 41, 327–350. doi: 10.1007/s10464-008-9165-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., and Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev. 82, 405–432. doi: 10.1111/j.1467-8624.2010.01564.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Dusenbury, L., Brannigan, R., Falco, M., and Hansen, W. B. (2003). A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ. Res. 18, 237–256. doi: 10.1093/her/18.2.237

PubMed Abstract | CrossRef Full Text | Google Scholar

Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., et al. (2010). Implementation fidelity of MyTeachingPartner literacy and language activities: association with preschoolers’ language and literacy growth. Early Child. Res. Q. 25, 329–347. doi: 10.1016/j.ecresq.2009.07.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Haymore-Sandholtz, J. (2002). Inservice training or professional development: contrasting opportunities in a school/university partnership. Teach. Teach. Educ. 18, 815–830. doi: 10.1016/S0742-051X(02)00045-8

CrossRef Full Text | Google Scholar

Horowitz, J. M., and Graf, N. (2021). Mobile Fact Sheet. Pew Research Center. Available at: https://www.pewresearch.org/internet/fact-sheet/mobile/

Google Scholar

Huntley, M. A. (2009). Measuring curriculum implementation. J. Res. Math. Educ. 40, 355–362. doi: 10.5951/jresematheduc.40.4.0355

CrossRef Full Text | Google Scholar

Jones, D. E., Greenberg, M., and Crowley, M. (2015). Early social emotional functioning and public health: the relationship between kindergarten social competence and future wellness. J. Public Health 105, 2283–2290. doi: 10.2105/AJPH.2015.302630

CrossRef Full Text | Google Scholar

Landry, S. H., Swank, P. R., Smith, K. E., Assel, M. A., and Gunnewig, S. B. (2006). Enhancing early literacy skills for preschool children: bringing a professional development model to scale. J. Learn. Disabil. 39, 306–324. doi: 10.1177/00222194060390040501

CrossRef Full Text | Google Scholar

Marsh, B., and Mitchell, N. (2014). The role of video in teacher professional development. Teach. Dev. 18, 403–417. doi: 10.1080/13664530.2014.938106

CrossRef Full Text | Google Scholar

Moffitt, T. E., Arseneault, L., Belsky, D., Dickson, N., Hancox, R. J., Harrington, H., et al. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proc. Natl. Acad. Sci. U. S. A. 108, 2693–2698. doi: 10.1073/pnas.1010076108

PubMed Abstract | CrossRef Full Text | Google Scholar

Mowbray, C. T., Holter, M. C., Teague, G. B., and Bybee, D. (2003). Fidelity criteria: development, measurement, and validation. Am. J. Eval. 24, 315–340. doi: 10.1177/109821400302400303

CrossRef Full Text | Google Scholar

Odden, A., Archibald, S., Fermanich, M., and Gallagher, H. A. (2002). A cost framework for professional development. J. Educ. Finance 28, 51–74.

Google Scholar

Pianta, R. C. (2005). “Standardized observation and professional development: a focus on individualized implementation and practices,” in Critical issues in early childhood professional development. eds. M. Zaslow and I. Martinez-Beck (Baltimore, MD: Paul H. Brookes Publishing), 231–254.

Google Scholar

Pianta, R. C., Mashburn, A. J., Downer, J. T., Hamre, B. K., and Justice, L. M. (2008). Effects of web-mediated professional development resources on teacher-child interactions in pre-kindergarten classrooms. Early Child. Res. Q. 23, 431–451. doi: 10.1016/j.ecresq.2008.02.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Quinn, D. M., and Kim, J. S. (2017). Scaffolding fidelity and adaptation in educational program implementation: experimental evidence from a literacy intervention. Am. Educ. Res. J. 54, 1187–1220. doi: 10.3102/0002831217717692

CrossRef Full Text | Google Scholar

Schonert-Reichl, K. A. (2017). Social and emotional learning and teachers. Futur. Child. 27, 137–155. doi: 10.1353/foc.2017.0007

CrossRef Full Text | Google Scholar

Shernoff, E. S., Lekwa, A. J., Reddy, L. A., and Coccaro, C. (2017). Examining teachers’ attitudes and experiences with coaching to inform research-based practice: an iterative developmental design study. J. Educ. Psychol. Consult. 27, 459–485. doi: 10.1080/10474412.2016.1255850

CrossRef Full Text | Google Scholar

Sklad, M., Diekstra, R., De Ritter, M., Ben, J., and Gravesteijn, C. (2012). Effectiveness of school- based universal social, emotional, and behavioral programs. Do they enhance students’ development in the area of skill, behavior, and adjustment? Psychology and Schools. 49, 892–909. doi: 10.1002/pits.21641

CrossRef Full Text | Google Scholar

Stains, M., and Vickrey, T. (2017). Fidelity of implementation: an overlooked yet critical construct to establish effectiveness of evidence-based instructional practices. CBE–life sciences. Education 16, 1–11. doi: 10.1187/cbe.16-03-0113

PubMed Abstract | CrossRef Full Text | Google Scholar

Taylor, R., Oberle, E., Durlak, J. A., and Weissberg, R. P. (2017). Promoting positive youth development through school-based social and emotional learning interventions: a meta-analysis of follow-up effects. Child Dev. 88, 1156–1171. doi: 10.1111/cdev.12864

PubMed Abstract | CrossRef Full Text | Google Scholar

van der Linden, S., van der Meij, J., and McKenney, S. (2022). Teacher video coaching, from design features to student impacts: a systematic literature review. Rev. Educ. Res. 92, 114–165. doi: 10.3102/00346543211046984

CrossRef Full Text | Google Scholar

Zhang, M., Lundenberg, M., Koehler, M., and Eberhardt, J. (2011). Understanding affordances and challenges of three types of video for professional development. Teacher Teacher Educ. 27, 454–462. doi: 10.1016/j.tate.2010.09.015

CrossRef Full Text | Google Scholar