Moving beyond dosage and adherence: A protocol for capturing dimensions of active child engagement as a measure of fidelity for social-emotional learning interventions

Social-emotional competencies are important for school-readiness and can be supported through social-emotional learning (SEL) interventions in the preschool years. However, past research has demonstrated mixed efficacy of early SEL interventions across varied samples, highlighting a need to unpack the black box of which early interventions work, under what conditions, and for whom. In the present article we discuss the critical implementation component of active child engagement in an intervention as a potential point of disconnect between the intervention as designed and as implemented. Children who are physically present but unengaged during an intervention may lead to decreased average impacts of an intervention. Furthermore, measuring young children’s active engagement with an intervention may help to guide iterative intervention development. We propose a four-step protocol for capturing the multi-dimensional and varied construct of active child engagement in a SEL intervention. To illustrate the utility of the protocol, we apply it to data from a pilot study of a researcher-implemented, semi-structured block play intervention focused on supporting the development of SEL and math skills in preschoolers. We then present future directions for the integration of active participant engagement into the measurement of implementation of SEL interventions for young children.

1. Introduction

Social-emotional competencies, including getting along with others, paying attention to and following directions, and regulating behaviors and emotions are crucial skills for children to develop prior to school entry as they predict later academic achievement and well-being (McClelland et al., 2007, 2013; Schmitt et al., 2017; Mackintosh and Rowe, 2021). However, interventions targeting social-emotional competencies in preschool have had mixed success, and scholars have recently suggested this may be due to issues with implementation fidelity, or perhaps, the way fidelity is typically measured (McClelland et al., 2017). For example, in the large majority of studies, implementation fidelity is typically assessed using simple measures of dosage (i.e., “Did they do it?”) and adherence (i.e., “Did it align with the guidelines?”). Active participant engagement (i.e., participation in the intervention) is seldom measured despite its inclusion in theoretical models of implementation (Century et al., 2010; Berkel et al., 2011). When active participant engagement in SEL interventions is included as a measure of fidelity, it almost exclusively concerns the active engagement of adults (e.g., teachers) with the intervention, and not child participant engagement.

Discounting how the autonomy of young children could lead to individual differences in children’s active participation during an intervention may lead to biased estimates of the efficacy of early SEL interventions. That is, a child may be physically present at an intervention session, but not actively engaging with the target material. Moreover, they may actively engage with only specific aspects or activities of the intervention but not others. These differences would not be captured by traditional measures of child-level dosage and could result in diminished effectiveness of an intervention as measured by estimates of average impacts. By assessing active child engagement as a multi-dimensional construct of implementation fidelity, we may be better able to capture individual differences in children’s experiences that moderate the effect of an intervention. Furthermore, considering the nuanced ways in which young children engage with an intervention may help researchers to iteratively develop interventions that support children from diverse backgrounds (e.g., considering differences by socio-cultural groups, site specific needs, etc.). Consequently, in the present article, we introduce a four-step protocol developed to capture active child engagement with SEL interventions. We use data from a pilot study of a semi-structured block play intervention (Schmitt et al., 2018a) as an example of applying the protocol to capture dimensions of active child engagement.

1.1. Supporting children’s early social-emotional learning through targeted interventions

Our broad definition of SEL interventions include intervention programs aimed at supporting the development of social skills, emotion regulation, and cognitive regulation (i.e., executive functions; McClelland et al., 2017). This definition is in line with the Collaborative for Academic Social and Emotional Learning (CASEL)’s framework that emphasizes the importance of competencies spanning social, emotional, and cognitive regulatory processes (CASEL, 2017). Several SEL interventions have been created to bolster these competencies in early childhood (e.g., McClelland et al., 2017; Nesbitt and Farran, 2021), a critical period in which rapid changes occur in cortical brain structures that are vital for SEL and cognitive development (Garon et al., 2008). This developmental period also coincides with children engaging with adults and peers in multiple settings (e.g., home, preschool; Schmitt et al., 2018a), in which SEL skills are necessary for successful relationships.

A common theme across the majority of early SEL interventions is that they are child-centered, which is thought to promote active child engagement with intervention content (Massouleh et al., 2012). For example, the Red Light, Purple Light intervention (Tominey and McClelland, 2011; Schmitt et al., 2015; McClelland et al., 2019) centers children’s experiences and active engagement by using fun and age-appropriate music and movement activities designed to promote behavioral self-regulation. During this intervention, children are given agency in interacting with the games as they choose and are also offered opportunities for autonomy in leading games. As another example, the Preschool Alternative THinking Strategies (PATHS) program offers explicit instruction in SEL through teacher-led lessons and extension activities like group games and art projects (Domitrovich et al., 2007). These activities were designed to be fun and engaging for young children while also promoting their SEL skills broadly. Brain Games (Barnes et al., 2021) is another classroom-based intervention for young children, focused on building attention, working memory, and inhibitory control through games.

Despite the fact that many SEL interventions take a child-centered approach, evidence of the efficacy of these interventions is mixed (McClelland et al., 2017; Nesbitt and Farran, 2021). Although some interventions like Red Light, Purple Light (Tominey and McClelland, 2011; Schmitt et al., 2015) and PATHS (Domitrovich et al., 2007) have shown positive effects on their target outcomes in preschoolers, other programs have shown mixed or null effects. For example, in a recent study Nesbitt and Farran (2021) found no positive impacts of the Tools of the Mind curriculum for supporting SEL in early childhood. Of note, Nesbitt and Farran were not the original developers of the Tools of the Mind curriculum. When researchers analyzed the implementation fidelity factors of adherence and dosage they found that intervention classrooms had varied dosage but implemented about half of what the developers expected, on average. However, neither the adherence to nor amount of time spent on the Tools curriculum was statistically related to children’s outcomes.

1.2. Integrated models of intervention implementation

To unpack the nuances of which early SEL interventions work under which conditions and for whom, it is essential to consider multiple factors of implementation fidelity. Theoretical frameworks of program implementation include factors controlled by the intervention designers (e.g., differentiation of intervention practices from currently enacted practices), factors controlled by the interventionist (e.g., script adherence and quality of delivery), and factors controlled by the participant (e.g., attendance and active engagement; Dane and Schneider, 1998; Carroll et al., 2007; Berkel et al., 2011). For example, Berkel et al.’s (2011) integrated model of program implementation differentiated behaviors that occur at the time of implementation that are directed by the interventionist from those directed by the participant (i.e., participant responsiveness). Furthermore, interventions can sometimes include multiple levels of implementation, such as when researchers train providers, who go on to train teachers or parents, who then teach children. Responsiveness to the intervention as intended includes attendance and dosage, retention, satisfaction, and active participant engagement, or participation in an intervention (Berkel et al., 2011). However, reviews of implemented educational interventions have demonstrated that interventionist-controlled factors (especially adherence) are more commonly reported than measures of participant responsiveness (O’Donnell, 2008). When responsiveness is considered, dosage is the most often-reported measure (Bos et al., 2022). Measurement of active participant engagement in early childhood interventions has primarily been concentrated on the adult participants who implement the intervention with children (e.g., teachers and parents’ active participation with the intervention). Some work has started to unpack the complex relations among fidelity indicators, including active participant engagement, at differing levels of an intervention. For example, Berkel et al. (2018) tested a theoretical cascade model in which facilitator delivery (e.g., adherence of the provider) predicted participant responsiveness (e.g., parents’ home practice), which in turn lead to improvements in the targeted outcomes (e.g., children’s mental health). However, very few studies have considered participant responsiveness fidelity indicators, such as active participant engagement, at the child-level of interventions designed to improve outcomes for young children.

2. Capturing dimensions of active child engagement

Given the lack of studies that report active child engagement—especially for young children—we sought to develop a general protocol for capturing young children’s active engagement with interventions as a measure of fidelity. The development of this protocol was guided by a conceptual framework comprised of the following assumptions:

• Individual differences in active child engagement may influence intervention efficacy.

• Contextual and individual factors may shape differences in active child engagement with an intervention.

• Active child engagement with an intervention can be measured through observing behavior.

• There are multiple dimensions to active child engagement with an intervention.

• There is variability in active child engagement with an intervention.

2.1. Individual differences in active child engagement may influence intervention efficacy

Active child engagement occurs within each intervention session, and thus “serves as a potential source of disconnect between the program as intended and the program as implemented” (Berkel et al., 2011, p. 23), making it a worthwhile focus of effort for measuring implementation fidelity. Dosage is a commonly reported measure of children’s participation in an intervention, but this fidelity indicator does not provide information about whether children experienced the intervention as it was intended. That is, children who are physically present but unengaged during intervention sessions may lead to decreased efficacy of the intervention as measured by estimates of average impacts. Therefore, it is important to consider how active child engagement may moderate the effect of an intervention.

2.2. Contextual and individual factors may shape differences in active child engagement with an intervention

Beyond considering how individual differences in engagement may influence intervention efficacy, it is also important to consider what factors predict differences in engagement with an intervention. School level factors such as discipline policies or classroom-level factors such as classroom culture and norms may affect active child engagement with a SEL intervention. For example, when schools and centers enact disciplinary measures that involve pulling children from an activity or classroom, they increase the chances of disrupting active child engagement. Classroom factors like the number of children in play groups have been associated with children’s positive engagement with peers, as smaller groupings promote more cooperative play (Howes et al., 2011). Intervention factors such as who is implementing the intervention (e.g., teachers or researchers) may also influence active child engagement. Finally, child level factors such as demographic background and baseline skills may influence children’s active engagement. By conducting in depth analyses of how children engage with an intervention as a measure of responsiveness, we can gather information that may challenge researchers’ assumptions about how children respond to materials and prompts. This is an especially important undertaking when working with historically under-represented populations in past research. This information may be used to inform iterative development of the intervention in an effort to create interventions that promote active child engagement for children from diverse backgrounds.

2.3. Active child engagement with an intervention can be measured through observing behavior

Although most research focused on active participant engagement has utilized participant report with adult participants (Carroll et al., 2007), engagement can be assessed through observing behavior, opening the door for assessment of engagement in young children. For example, Ling and Barnett (2013) used behavior observations time-sampled in 15-s intervals to assess groups of preschoolers’ engagement in circle time activities. Another researcher-developed measure is the inCLASS (Downer et al., 2010), which is an observational tool focused on capturing preschoolers’ interactions with teachers, peers, and classroom activities. In the context of early SEL interventions, Schmitt et al. (2018b) used a similar measure to assess preschoolers’ on-task behavior during the Positive Action intervention, aimed at improving social-emotional competence and health behaviors. One efficacy study of the Tools of the Mind intervention also included a measure of child engagement with the intervention using a Likert scale based on observations of child behavior ranging from completely off-task to intense focus across time-sampled intervals (Child Observation in Preschool, COP; Nesbitt and Farran, 2021).

2.4. There are multiple dimensions to active child engagement with an intervention

The aforementioned work that measured and reported active child engagement in interventions approached it as a unidimensional construct and did not consider the complex nature of the intervention environment in which young children often have varied points of opportunity for engagement with an intervention as intended. For example, a child may appear to be actively engaged in an intervention activity (e.g., by being socially engaged with their peers) without engaging with the targeted SEL content. As another example, in a randomized trial of the Red Light, Purple Light intervention, the authors describe that although “the majority of children actively participated in all of the playgroup games… a few children chose to watch on occasion” (Tominey and McClelland, 2011 p. 513). Although these children were actively engaged with the content of the games by watching, they were not actively engaged with the physical movements of the games, a separate dimension of engagement. By considering active child engagement as a unidimensional construct, we miss capturing varied levels of engagement with the intervention, which may provide biased estimates of program outcomes. Furthermore, considering multiple dimensions of active child engagement allow for the ability to test which child participant-involved components are most important for growth in the target outcomes. Thus, a key extension of existing research is our consideration of active child engagement with an intervention as a multi-dimensional construct.

2.5. There is variability in active child engagement with an intervention

A final assumption behind our approach to measuring active child engagement is that it is varied, both between and within children. Decades of research in cognitive developmental psychology have converged to support the theory that development is defined by variability (Siegler, 2007). That is, differences in children’s thinking and behavior exist between individual children, within the same child in different contexts, and even within the same child in the same context at a different point in time (Siegler, 1994). We believe that it is important to take this variability into account when approaching the measurement of active child engagement with an intervention. For example, a child’s engagement may vary across intervention sessions in response to the content, or across the course of a single session. By operationalizing engagement in a way that takes variability into account, researchers can capture nuances such as the “implementation dip,” or a decrease in performance or adherence in response to change (Fullan, 2001) in the context of active child engagement with SEL interventions.

3. Four-step protocol for capturing dimensions of active child engagement

In line with past theoretical frameworks that use a multi-step approach as a solution for measuring the complex construct of fidelity (e.g., Hulleman et al., 2013), we propose a four-step protocol for capturing active child engagement:

1. Identify points of opportunity for active child engagement with the intervention to specify multiple dimensions of active child engagement.

2. Operationalize and measure the dimensions of active child engagement.

3. Analyze the dimensions of active child engagement.

4. Link the dimensions of active child engagement to other variables.

3.1. Applying the four-step protocol to measure preschoolers’ active engagement with the Block Play Intervention

In this section, we use data from a pilot study of the Block Play Intervention to illustrate application of the four-step protocol. The Block Play Intervention is a brief, semi-structured, play-based intervention aimed at supporting preschoolers’ cognitive regulation, a critical component of SEL (McClelland et al., 2017), and mathematics through small group interactions. The intervention includes twice-weekly sessions of small group play (two to three children) with wooden unit blocks. Children are given specific building goals at the start of the session by a researcher interventionist (e.g., “Today your job is to build a tower together!”) but are then allowed to build freely. Over the course of the intervention sessions, the prompts gradually become more complex so that children’s cognitive regulation is challenged, not just used (e.g., Week five: “Today your job is to build a house together… It needs to have four walls, a roof, a way to get inside like a door, and at least two rooms”; Week seven: “Today I am going to show you a picture of a structure. Your job is to work together to build the structure you see in this picture.”). These prompts were targeted at priming children to work together and engage in social interaction to build structures in collaboration with their groupmates while also building their self-regulation skills through avoiding distractions and engaging in goal-oriented behavior.

A pilot study of this intervention included a sample of 59 children (M_age = 55.20 months), randomly assigned to the intervention group (n = 24) or a business as usual (BAU; n = 35) control group. On average, children in the intervention condition participated in 13 of the 14 sessions, with a range of 11 to 14 sessions attended. Researcher interventionists adhered to the building prompt scripts 94% of the time. The application of the four-step protocol for capturing active child engagement to the pilot data of the Block Play Intervention is presented visually in Figure 1. We use the context of this researcher-implemented pilot study as a straightforward example of the protocol’s application but believe the four-step protocol can be scaled and utilized with much larger and more complex SEL interventions with young children.

FIGURE 1

Figure 1. Applying the four-step protocol to measure preschoolers’ active engagement with the Block Play Intervention.

3.1.1. Step 1: Identify points of opportunity for engagement to specify dimensions of active child engagement

The first step is to identify opportunities for potential active child engagement with the intervention. As with all aspects of fidelity, the identification of active child engagement opportunities should be closely tied to the intervention’s overarching theory of change (Darrow, 2013). It may be helpful to think of these points of opportunity as participant-involved core intervention components, or the aspects of the intervention in which child participants are directly involved that are theorized to lead to change in the child-level outcome variables. The theory of change of the Block Play Intervention is presented in Figure 2.

FIGURE 2

Figure 2. Theory of change for the Block Play Intervention.

In the Block Play Intervention, we hypothesized that participation in semi-structured block play sessions would lead to gains in the outcomes of interest through three points of potential active child engagement: the block play itself, positive interactions with peers, and working toward a provided, increasingly complex goal (see Row 1 of Figure 1). Block play provides children with opportunities to practice working with abstract concepts and representations, which may help to develop cognitive regulation (Wolfgang et al., 2001; Hadani and Rood, 2018). Children also get the opportunity to practice fine motor skills through block play, which relate to cognitive regulation and mathematical cognition development in the early years (Gashaj et al., 2019; McClelland and Cameron, 2019). Working cooperatively with peers by negotiating and engaging in prosocial behaviors may help children to develop their language and interpersonal skills (Sluss and Stremmel, 2004), as well as emotional and behavioral self-regulation skills like inhibiting prepotent responses (e.g., knocking down a peers’ tower when angry). Finally, the goal-directed aspect of the activity may also help to strengthen children’s cognitive regulation and academic skills as children are required to utilize metacognitive resources to remember and plan their building behavior in accordance with the goal (Schmitt et al., 2018a).

After identifying these points of opportunity for potential engagement, we used them to generalize and name the multiple dimensions of participant engagement that we wished to capture. For the Block Play Intervention, this resulted in three general dimensions of child engagement: Engagement with the activity/materials, engagement with peers, and engagement with the learning goal (see Row 2 of Figure 1). Although our specified dimensions may apply to many play-based or semi-structured early SEL interventions, researchers wishing to use this protocol can alter or substitute these dimensions as needed. For example, SEL interventions that include teacher or parent interactions may need to add a dimension for children’s engagement with a caregiver. Furthermore, researchers testing more comprehensive interventions may wish to draw the points of opportunity for potential engagement from the specific activities in the intervention or create multiple levels (e.g., engagement with materials and peers within specific intervention activities like shared book reading, role playing, or group games).

3.1.2. Step 2: Operationalize and measure dimensions of active child engagement

The second step is to operationalize each dimension of active child engagement (see Row 3 in Figure 1) so that it can be quantified. Measurement decisions should be guided by pilot work, past literature, and the intervention theory of change. Researchers may choose to use previously developed measures of child engagement (e.g., inCLASS, Downer et al., 2010; COP, Farran, 2011) or to develop their own observational measures of specific behaviors.

3.1.2.1. Engagement with the activity/materials in the Block Play Intervention

Aligned with our conceptual framework, we chose to operationalize each dimension of child engagement with the Block Play Intervention through observation of behavior captured on video recordings of the 15-min block play sessions. For the engagement with activity/materials dimension, we used time-sampled observations of “on-task” behavior (Ling and Barnett, 2013). Coders watched videos and documented codes for each child individually, replaying sections of the video as needed. As past pilot work suggested that children would engage in on-task block play behavior for much of the session, we chose to use momentary time-sampling of whether or not children were actively engaged in on-task block play behavior after every minute of play. That is, it was important to use a higher frequency of coding instances (after every minute) to adequately capture variability and examine individual differences in children’s engagement with the activity and materials. Children received a score of 0–15 for each session.

3.1.2.2 Engagement with peers in the Block Play Intervention

When operationalizing the engagement with peers dimension, we chose to focus on positive peer interaction behaviors as they were hypothesized to be related to gains in the outcome variables. We drew the positive peer interaction behaviors of interest from two sub-scales of the Minnesota Preschool Affect Checklist Revised/Shortened (M-PAC-R/S; Denham et al., 2012)–the Leading and Joining and the Empathy and Prosocial Behavior sub-scales. We coded in 5-min whole-interval time-sampled increments (a coarser level of analysis than the engagement with activity/materials dimension) in line with the protocol for using the M-PAC-R/S. Children therefore received a score of 0–3 for each session corresponding to positively engaging with peers during none of the three 5-min intervals (0) to positively engaging with peers during all of the three 5-min intervals (3). To code, we determined if a child engaged in any of these five behaviors at any time in the prior 5 min: successfully leading an activity, successfully joining an activity, facilitating turn-taking, cooperating with a peer or group to achieve a common goal, or exhibiting sharing behavior. For example, consider this dyad of children who were given the goal of building a castle for a king and queen. In the process, they engaged in leading and joining behaviors, taking turns in adding blocks to the structure, and working together cooperatively:

Child A: [Pointing to blocks] “So this is the king, that’s the queen. King, queen. And we need a bottom slip. That will help… So that will go right here.”

Child B: “Put it right on this.”

Child A: “We need another one of these pieces.”

Child B: “… I found one!”

A different dyad of children took another cooperative building approach, where one child led the building activity, and another gathered and shared materials:

Child C: “Wait, wait… I need baby triangles… I need another triangle… A baby one. Will you find it for me?”

Child D: “I will find it for you.”

Child C: [taking triangle from Child D] “This goes here.”

3.1.2.3 Engagement with learning goals in the Block Play Intervention

Finally, we operationalized engagement with the learning goal as the proportion of the session a child spent engaged with the explicit building goal given to them by the interventionist, as evidenced by verbal and non-verbal behaviors. This dimension was also coded in whole-interval 5-min increments by determining whether a child spent any of the time in the previous 5 min building the assigned structure for the session. Children therefore received a score of 0–3 for each session corresponding to engaging with the building goal during none of the three 5-min intervals (0) to engaging with the building goal during all of the three 5-min intervals (3). Examples of children who did not engage with the building goal varied. For example, when asked to build a tower, one child responded, “No thank you. We are making a playground.” Other children chose not to engage with the given building goal, and instead requested to be able to build freely. For example, when children were asked to model their structure after a picture, one child responded, “I’m not gonna build that. Can I just build?”

For each dimension of engagement, we calculated the proportion of the session spent engaged in the specified behaviors of that dimension. That is, for the engagement with activity/materials dimension, we calculated the proportion of minutes children were engaged in on-task block play behavior out of the total 15 min. For the engagement with peers and learning goals dimensions, we calculated the proportion of intervals children were engaged in either leading, joining, taking turns, working cooperatively and sharing behaviors or goal-oriented behaviors out of the total three coded intervals. Proportions were averaged across the 14 sessions to create an overall engagement score for each dimension throughout the course of the intervention. As we were also interested in considering within-session variability, we also averaged across sessions to create a variable of average engagement in each dimension for the first 5 min, middle 5 min, and final 5 min of the 15-min sessions.

3.1.3. Step 3: Analyze dimensions of active child engagement

The third step of the protocol is to intensively analyze the dimensions of engagement variables. The overarching goal of this step is to answer the questions of whether children actively engaged in the intervention as intended across dimensions, and how their engagement in each dimension varied within and across sessions. See Row 4 of Figure 1 (Step 3) for suggestions on how to analyze these variables.

3.1.3.1. Engagement with the activity/materials in the Block Play Intervention

Data was converted to percentages of the calculated proportions for interpretation, but descriptive statistics of the raw data by session are presented in Table 1. Children in the intervention condition were engaged with on-task block building activity 81.0% of the time on average. There was variability between individuals, with the least engaged child engaging in on-task behavior an average of 58.5% of the time and the most engaged child engaging in on-task behavior an average of 95.2% of the time. Examining within-individual variability also provided valuable information about whether the design of the intervention was able to sustain children’s engagement in on-task block play over time. Children were fairly consistent with their engagement over the course of the 15-min sessions, although there was some decline; they engaged in on-task behavior 84.4% of the time, on average, in the first 5 min of sessions, 81.4% of the time in the next 5 min, and 73.6% of the time in the last 5 min. There was also some variability in engagement with the activity/materials across sessions (see Figure 3), but there was not an overall decline in engagement with block play as the intervention went on. Overall, our analysis of this dimension suggests that although on-task behavior was high, there was between-individual variability which could potentially influence the strength of efficacy of the intervention. Furthermore, our design of the block play sessions seems to have been successful at keeping children actively engaged in on-task behavior across the intervention sessions.

TABLE 1

Table 1. Descriptive statistics by session.

FIGURE 3

Figure 3. Engagement with the activity/materials across sessions of the Block Play Intervention. Figure shows percentage of engagement out of total coded time (15 min) with the activity/materials at each session. The range from minimum to maximum engagement is represented by error bars.

3.1.3.2. Engagement with peers in the Block Play Intervention

Descriptive statistics of the raw data by session are presented in Table 1. Children were engaged in positive peer interactions an average of 57.0% of the intervals sampled. There was substantial variability between individual children, with the least engaged child only engaged in positive interactions with peers an average of 12.0% of the time, and the most engaged child engaged an average of 97.7% of the time. Children were most engaged in positive interactions with peers near the start of the session (61.5% in the first 5 min, on average), which declined over the course of the session (54.2% in the next 5 min and 50.0% in the last 5 min, on average). There was also considerable variability in children’s engagement with peers across sessions (see Figure 4). However, there was not an overall pattern of children positively engaging with peers more or less as the intervention went on.

FIGURE 4

Figure 4. Engagement with peers across sessions of the Block Play Intervention. Figure shows percentage of engagement out of total coded time (three time intervals) with peers at each session. The range from minimum to maximum engagement is represented by error bars.

Overall, analyses of the Engagement with Peers dimension highlighted that despite the prompts for children to work together on their building, children only positively engaged with peers an average of 57% of the sampled intervals. As cooperative play typically emerges between 4 and 5 years of age (Parten, 1932), this level of peer interaction seems developmentally appropriate. Children were more likely to engage positively with peers just after the building prompt was given (in the first 5 min), suggesting that adding scaffolding of peer interactions or reminders to work together throughout the course of the sessions in future iterations of the intervention may increase positive peer engagement.

3.1.3.3 Engagement with learning goals in the Block Play Intervention

Descriptive statistics of the raw data by session are presented in Table 1. Children were engaged with the provided building goal an average of 77.0% of the intervals. There was variability between the least engaged child (39%, on average) and the most engaged child (100%). Again, children’s engagement with the goal generally declined over the course of the session, with a drop-off near the end of the sessions (M = 84.3% in first 5 min; 80.0% in second 5 min; 63.9% in the last 5 min). There was some variability in goal engagement across the course of the intervention (see Figure 5). With the exception of one session early on, engagement with the goal was lowest for the most complex building goals, in which children were required to copy a model of a sophisticated structure, given in the final four sessions.

FIGURE 5

Figure 5. Engagement with learning goals across sessions of the Block Play Intervention. Figure shows percentage of engagement out of total coded time (three time intervals) with the learning goal at each session. The range from minimum to maximum engagement is represented by error bars.

3.1.3.4. Engagement across dimensions in the Block Play Intervention

To demonstrate multi-dimensionality of engagement with the Block Play Intervention for the 24 children in the pilot intervention condition, we standardized the dimensions of engagement variables and have presented them as a heat map in Figure 6. Each column represents an individual child, with increasingly above average engagement in a specific dimension represented as a darkening shade. Below average engagement is indicated with lighter shades. Some children were uniformly engaged or disengaged with the intervention across the dimensions. For example, Participant 1 was above the average of the sample in their engagement with the activity/materials, with peers, and with the learning goal. Conversely, Participant 17 was below average in their engagement across all dimensions. However, other children showed more nuanced engagement with different aspects of the intervention. Participant 16 was highly positively engaged with their peers compared to the rest of the sample but was below average in their engagement with the activity and the goal. Specifically, this child was frequently distracted from building (e.g., walking around the room) and when they did engage in building, they preferred to set their own goals. However, they were highly social with peers, offering positive encouragement and engaging in sharing behaviors during play. Participant 19 was engaged with block play activity in alignment with the provided building goal but was not as positively engaged with peers as the rest of the sample, as this child preferred to build alone. When a peer did attempt to engage with them, the interactions were often negative (e.g., refusing to let another child help with the structure). Finally, Participant 13 was highly engaged with block play and with peers compared to the rest of the sample but was slightly below average in their engagement with the provided building goal. These results demonstrate that, even in this small sample, children were differentially engaged with the multiple points of opportunity for engagement. By separating the multiple dimensions of engagement, we can test the most important dimensions of engagement for intervention efficacy in the fourth step of the protocol.

FIGURE 6

Figure 6. Heat map of individual children’s multi-dimensional active engagement with the Block Play Intervention. Figure shows individual children’s engagement with activity/materials, peers, and learning goals across the Block Play Intervention. Each column represents an individual child, and the rows illustrate that child’s engagement with a specific dimension, relative to other children in the sample. Darker shades demonstrate that a child was increasingly above average, compared to the rest of the sample, on a dimension of engagement. Lighter shades demonstrate that a child was increasingly below average on a dimension of engagement.

3.1.4. Step 4: Link active child engagement to other measures

In the final step of the protocol, the dimensions of active child engagement should be linked to other variables. First, researchers can explore the assumption that differences in active child engagement can be shaped by contextual and individual factors, like school-level, demographic or baseline skill variables. Next, it is important to link the dimensions of active child engagement to outcome measures to assess how active child engagement may influence intervention effectiveness.

3.1.4.1 Individual predictors of active child engagement in the Block Play Intervention

To consider individual-level predictors of active child engagement with the Block Play Intervention, we estimated correlations among the dimensions of engagement and the demographic variables of age and parent education level, as well as children’s baseline social skills and problem behaviors as rated by their classroom teacher using the Social Skills Improvement System rating scale (Gresham and Elliott, 2008; see Table 2). Teachers were asked to use the scale to rate children’s frequency of behaviors in social skills including communication and cooperation and problem behaviors including hyperactivity/inattention. We also calculated an overall engagement score by summing the standardized scores of engagement across each dimension. Results of these correlations should be interpreted cautiously, given the small sample size. Age was not significantly correlated with any dimension or overall engagement (all p values > 0.05). However, parent education was significantly correlated with overall engagement (r = 0.54, p = 0.01), engagement with activity/materials (r = 0.43, p = 0.04), and engagement with peers (r = 0.48, p = 0.02). Parent education and engagement with the learning goals were not significantly correlated at r = 0.31, p = 0.16.

TABLE 2

Table 2. Descriptive statistics and correlations among dimensions of engagement and variables of interest for Step 4 of the protocol from the Block Play Intervention.

Teacher-ratings of children’s social skills were also significantly correlated with overall engagement (r = 0.40, p = 0.04), engagement with the activity/materials (r = 0.46, p = 0.03) and with the learning goals (r = 0.60, p = 0.002). Although the correlation between baseline social skills and engagement with peers was not statistically significant, it was positive (r = 0.11, p = 0.57). Conversely, teacher-rated child problem behaviors were significantly negatively correlated with overall engagement (r = −0.55, p = 0.01), engagement with the activity/materials (r = −0.56, p = 0.01), and with goals (r = −0.76, p < 0.001). Again, the correlation with the engagement with peers dimension was not statistically significant, but was negative (r = −0.26, p = 0.32). Overall, these analyses revealed interesting insights important for future iterative development of the intervention. First, although previous work found that the Block Play Intervention was particularly efficacious for children whose parents had less educational attainment (Schmitt et al., 2018a) we found that children whose parents have higher educational attainment were more actively engaged with the intervention than their peers. This finding highlights a need to unpack potential reasons and explore additional supports for increasing active child engagement in future iterations as we strive to create equitable interventions appropriate for children from diverse backgrounds. We also found that children who were rated as engaging in more problem behaviors in the classroom by their teacher were less engaged with the intervention, and particularly with the learning goal. Again, in future iterations of the Block Play Intervention, we hope to explore how to scaffold these children’s goal-orientation.

3.1.4.2 Linking active child engagement to outcome measures in the Block Play Intervention

Next, we moved to considering how children’s overall active engagement and engagement with the activity/materials, peers, and learning goals in the Block Play Intervention related to gains in self-regulation and math outcomes. We calculated pre-to post-gain scores in three outcomes of interest: behavioral self-regulation, measured by the Head-Toes-Knees-Shoulders task (HTKS; McClelland et al., 2014); cognitive flexibility, measured by the Three-Dimensional Change Card Sort Task (DCCS; Zelazo, 2006); and mathematics-specific language, measured by the researcher-developed Math Language Assessment (Purpura and Logan, 2015). See Schmitt et al. (2018a) for assessment details. We estimated correlations among these gain scores and children’s active engagement with the activity/materials, peers, and learning goals, as well as overall engagement (see Table 2). Four children were missing post-test data on these measures. None of the dimensions of engagement or overall engagement were significantly correlated with the gain scores (all p values > 0.05). Despite this, the pattern of correlation coefficients revealed interesting associations among active child engagement and growth in the outcome variables. For example, overall engagement with the intervention was similarly related to gains in behavioral self-regulation (r = 0.20, p = 0.39) and cognitive flexibility (r = 0.19, p = 0.44). However, when the dimensions of engagement variables were considered separately, gains in behavioral self-regulation were similarly related to engagement with the activity/materials (r = 0.18, p = 0.44), peers (r = 0.19, p = 0.43), and goals (r = 0.17, p = 0.49), but gains in cognitive flexibility were more related to active child engagement with peers (r = 0.39, p = 0.09) than with the activity/materials (r = 0.06, p = 0.71) or goals dimensions (r = −0.03, p = 0.91). Gains in mathematics-specific language may also be more related to one dimension of engagement with the intervention (activity/materials, r = 0.22, p = 0.35) than the others (peers, r = 0.08, p = 0.75; goals, r = −0.04, p = 0.74). Although these correlation analyses are underpowered, the pattern of results supports our theoretical assumptions that active child engagement with an intervention is multi-dimensional, and engagement with specific dimensions may be particularly important for supporting different outcomes.

In larger samples, along with traditional measures of fidelity (e.g., dosage and adherence) the separate dimensions of engagement should be considered as moderators of treatment effects on outcome measures. For example, in future iterations of the Block Play Intervention, we plan to test dimensions of engagement as moderators of the effect of condition. We will also test interactions among these dimensions and hypothesize that children who are highly engaged across dimensions will benefit the most from the intervention.

4. Conclusion and future directions

Despite theoretical models that include multiple factors of participant responsiveness, the crucial aspect of active participant engagement is often overlooked, especially for young child participants. Some early SEL interventions (e.g., Nesbitt and Farran, 2021) have mixed evidence of efficacy but measuring active engagement with an intervention at the child level may help the field to unpack these mixed results. Furthermore, the multi-dimensionality and variability of children’s active engagement with the intervention must be considered to capture children’s experiences in the complex intervention environment. In this article, we have presented a four-step protocol for identifying, operationalizing, and analyzing children’s active engagement with multiple dimensions of an intervention. We encourage fellow researchers to prioritize incorporating participant engagement at the child level into the measurement of implementation fidelity in SEL interventions with young children and conclude by presenting suggestions to further this work. Although this protocol has been initially applied to a pilot study of a researcher-implemented intervention with a small sample size, we believe it can be scaled for use with larger and more comprehensive intervention studies.

In the fourth step of the protocol, we suggested that future work using this protocol may wish to explore contextual factors that predict the dimensions of engagement. For example, research specifically focused on supporting SEL through play-based learning would benefit from the examination of play as it is defined within different cultural contexts. Children’s play-based learning experiences are scaffolded by their culture and given the impact of globalization, successful interventions would require consideration of the different cultural belief systems of the children and families included in the study (Harkness and Super, 1993). Children have different exposure to play activities within the home environment which affects the variety of play experiences that children are exposed to and possibly their level of active engagement in various dimensions of an intervention activity. Future work is also needed to consider how individual children’s engagement may affect peers’ engagement, especially during group work.

Our example, the Block Play Intervention, did not include an active control condition, but researchers that include a closely aligned active control condition may wish to code active child engagement with the active control condition. For example, in another iteration of the Block Play Intervention, we plan to test the efficacy of the semi-structured play sessions against a free block play condition, in which children are not given specific building goals. We will code and compare engagement with the activity and with peers to compare to the semi-structured condition to test whether there are differences in engagement by condition and if individual differences in engagement relate to gains in the target outcomes.

We focused this article on child participants’ engagement in interventions, but future work is needed to consider the bi-directional relations between interventionist and child-directed factors of implementation. For example, theoretical models posit that interventionist-directed behaviors influence participants’ responsiveness to an intervention (e.g., a cascade model; Berkel et al., 2018). However, child engagement across the dimensions likely also influences interventionist behaviors (Berkel et al., 2011). That is, interventionist adherence to a script and quality of delivery is likely influenced by children’s engagement in the activity at hand, as the interventionist responds to the children’s behavior. The Block Play Intervention was a researcher-implemented intervention, but researchers testing teacher-implemented SEL interventions may also include measures of active teacher participant engagement to consider relations among teacher and child engagement variables. Untangling and determining the best ways to capture the interaction between these factors is an important next step for applying integrated theoretical models of program implementation to practice.

Finally, protocols are also needed to guide the nuanced measurement and analysis of other understudied aspects of fidelity. For example, quality of the delivery of an intervention also includes multiple dimensions and is varied between and within interventionists. Intervention work with school-aged children has explored some of the dimensions of this domain (e.g., practice opportunities, modeling, feedback, scaffolding; Doabler et al., 2021). Additional work is needed to unpack quality in interventions designed for young children and to create general protocols for guiding the identification, operationalization, and analyses of these variables.

In summary, we hope to encourage SEL intervention researchers to consider active child engagement as a worthwhile area of focus in the measurement of implementation fidelity. We have introduced the four-step protocol as a general guide for capturing dimensions of active child engagement. In doing so, the field may be better able to discover which early SEL interventions work, under which conditions, and for whom.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Purdue University Institutional Review Board. Written informed consent to participate in this study was provided by the participants’ legal guardian/next of kin.

Author contributions

BD conceptualized the protocol, conducted analyses, interpretation of data, and led writing of manuscript. BD, TP, EG, and LB wrote the manuscript. TZ, IK, and KM conducted data coding and analysis. SS conceived of original study, directed data collection, provided input on conceptualization of protocol, and interpretation of data. All authors provided critical feedback and approved the submitted version.

Funding

This work was supported by the National Science Foundation, Award #2000641.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Barnes, S. P., Bailey, R., and Jones, S. M. (2021). Evaluating the impact of a targeted approach designed to build executive function skills: A randomized trial of brain games. Front. Psychol. 12:655246. doi: 10.3389/fpsyg.2021.655246

PubMed Abstract | CrossRef Full Text | Google Scholar

Berkel, C., Mauricio, A. M., Sandler, I. N., Wolchik, S. A., Gallo, C. G., and Brown, C. H. (2018). The cascading effects of multiple dimensions of implementation on program outcomes: A test of a theoretical model. Prev. Sci. 19, 782–794. doi: 10.1007/s11121-017-0855-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Berkel, C., Mauricio, A. M., Schoenfelder, E., and Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prev. Sci. 12, 23–33. doi: 10.1007/s11121-010-0186-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Bos, S. E., Powell, S. R., Maddox, S. A., and Doabler, C. T. (2022). A synthesis of the conceptualization and measurement of implementation fidelity in mathematics intervention research. J. Learn. Disabil. 222194211065498. doi: 10.1177/00222194211065498

PubMed Abstract | CrossRef Full Text | Google Scholar

Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., and Balain, S. (2007). A conceptual framework for implementation fidelity. Implement. Sci. 2:40. doi: 10.1186/1748-5908-2-40

PubMed Abstract | CrossRef Full Text | Google Scholar

CASEL (2017). What is the CASEL framework? Chicago, IL: CASEL.

Google Scholar

Century, J., Rudnick, M., and Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. Am. J. Eval. 31, 199–218. doi: 10.1177/1098214010366173

CrossRef Full Text | Google Scholar

Dane, A. V., and Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clin. Psychol. Rev. 18, 23–45. doi: 10.1016/S0272-7358(97)00043-3

CrossRef Full Text | Google Scholar

Darrow, C. L. (2013). The effectiveness and precision of intervention fidelity measures in preschool intervention research. Early Educ. Dev. 24, 1137–1160. doi: 10.1080/10409289.2013.765786

CrossRef Full Text | Google Scholar

Denham, S. A., Bassett, H. H., Thayer, S. K., Mincic, M. S., Sirotkin, Y. S., and Zinsser, K. (2012). Observing preschoolers’ social-emotional behavior: Structure, foundations, and prediction of early school success. J. Genet. Psychol. 173, 246–278. doi: 10.1080/00221325.2011.597457

PubMed Abstract | CrossRef Full Text | Google Scholar

Doabler, C. T., Clarke, B., Kosty, D., Fien, H., Smolkowski, K., Liu, M., et al. (2021). Measuring the quantity and quality of explicit instructional interactions in an empirically validated tier 2 kindergarten mathematics intervention. Learn. Disabil. Q. 44, 50–62. doi: 10.1177/0731948719884921

CrossRef Full Text | Google Scholar

Domitrovich, C. E., Cortes, R. C., and Greenberg, M. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. J. Prim. Prev. 28, 67–91. doi: 10.1007/s10935-007-0081-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Downer, J. T., Booren, L. M., Lima, O. K., Luckner, A. E., and Pianta, R. C. (2010). The individualized classroom assessment scoring system (inCLASS): Preliminary reliability and validity of a system for observing preschoolers’ competence in classroom interactions. Early Child. Res. Q. 25, 1–16. doi: 10.1016/j.ecresq.2009.08.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Farran, D. C. (2011). Child observation in preschool (With adaptations for use in the Tools of the Mind curriculum evaluation). Nashville, TN: Peabody Research Institute, Vanderbilt University.

Google Scholar

Fullan, M. (2001). The new meaning of educational change. London: Routledge.

Google Scholar

Garon, N., Bryson, S. E., and Smith, I. M. (2008). Executive function in preschoolers: A review using an integrative framework. Psychol. Bull. 134, 31–60. doi: 10.1037/0033-2909.134.1.31

PubMed Abstract | CrossRef Full Text | Google Scholar

Gashaj, V., Oberer, N., Mast, F. W., and Roebers, C. M. (2019). The relation between executive functions, fine motor skills, and basic numerical skills and their relevance for later mathematics achievement. Early Educ. Dev. 30, 913–926. doi: 10.1080/10409289.2018.1539556

CrossRef Full Text | Google Scholar

Gresham, F. M., and Elliott, S. N. (2008). Social skills improvement system: Rating scales. London: Pearson Assessments.

Google Scholar

Hadani, H. S., and Rood, E. (2018). The roots of STEM success: Changing early learning experiences to build lifelong thinking skills. San Francisco, CA: Center for Childhood Creativity.

Google Scholar

Harkness, S., and Super, C. M. (1993). “The developmental niche: Implications for children’s literacy development,” in Early intervention and culture: Preparation for literacy, eds L. Eldering and P. Lesemen (Paris: UNESCO), 115–132.

Google Scholar

Howes, C., Guerra, A. W., Fuligni, A., Zucker, E., Lee, L., Obregon, N. B., et al. (2011). Classroom dimensions predict early peer interaction when children are diverse in ethnicity, race, and home language. Early Child. Res. Q. 26, 399–408.

Google Scholar

Hulleman, C. S., Rimm-Kaufman, S. E., and Abry, T. (2013). “Innovative methodologies to explore implementation: Whole-part-whole—construct validity, measurement, and analytical issues for intervention fidelity assessment in education research,” in Applying implementation science in early childhood programs and systems, eds T. Halle, A. Metz, and I. Martinez-Beck (Baltimore, MD: Paul H Brookes Publishing), 65–93.

Google Scholar

Ling, S. M., and Barnett, D. W. (2013). Increasing preschool student engagement during group learning activities using a group contingency. Topics Early Child. Spec. Educ. 33, 186–196. doi: 10.1177/0271121413484595

CrossRef Full Text | Google Scholar

Mackintosh, B. B., and Rowe, M. (2021). Baseline inequalities: Social skills at preschool entry moderate math development. J. Res. Child. Educ. 35, 1–21. doi: 10.1080/02568543.2020.1728446

CrossRef Full Text | Google Scholar

Massouleh, N. S., Branch, R., and Jooneghani, R. B. (2012). Learner-centered instruction: A critical perspective. J. Educ. Pract. 3, 50–59.

Google Scholar

McClelland, M. M., Acock, A. C., Piccinin, A., Rhea, S. A., and Stallings, M. C. (2013). Relations between preschool attention span-persistence and age 25 educational outcomes. Early Child. Res. Q. 28, 314–324. doi: 10.1016/j.ecresq.2012.07.008

PubMed Abstract | CrossRef Full Text | Google Scholar

McClelland, M. M., and Cameron, C. E. (2019). Developing together: The role of executive function and motor skills in children’s early academic lives. Early Child. Res. Q. 46, 142–151.

Google Scholar

McClelland, M. M., Cameron, C. E., Connor, C. M., Farris, C. L., Jewkes, A. M., and Morrison, F. J. (2007). Links between behavioral regulation and preschoolers’ literacy, vocabulary, and math skills. Dev. Psychol. 43, 947–959. doi: 10.1037/0012-1649.43.4.947

PubMed Abstract | CrossRef Full Text | Google Scholar

McClelland, M. M., Cameron, C. E., Duncan, R., Bowles, R. P., Acock, A. C., Miao, A., et al. (2014). Predictors of early growth in academic achievement: The head-toes-knees-shoulders task. Front. Psychol. 5:599. doi: 10.3389/fpsyg.2014.00599

PubMed Abstract | CrossRef Full Text | Google Scholar

McClelland, M. M., Tominey, S. L., Schmitt, S. A., and Duncan, R. (2017). SEL interventions in early childhood. Future Child. 27, 33–47.

Google Scholar

McClelland, M. M., Tominey, S. L., Schmitt, S. A., Hatfield, B. E., Purpura, D. J., Gonzales, C. R., et al. (2019). Red light, purple light! Results of an intervention to promote school readiness for children from low-income backgrounds. Front. Psychol. 10:2365. doi: 10.3389/fpsyg.2019.02365

PubMed Abstract | CrossRef Full Text | Google Scholar

Nesbitt, K. T., and Farran, D. C. (2021). Effects of prekindergarten curricula: Tools of the mind as a case study. Monogr. Soc. Res. Child Dev. 86, 7–119.

Google Scholar

O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Rev. Educ. Res. 78, 33–84.

Google Scholar

Parten, M. B. (1932). Social participation among pre-school children. J. Abnorm. Soc. Psychol. 27, 243–269. doi: 10.1037/h0074524

CrossRef Full Text | Google Scholar

Purpura, D. J., and Logan, J. A. R. (2015). The nonlinear relations of the approximate number system and mathematical language to early mathematics development. Dev. Psychol. 51, 1717–1724. doi: 10.1037/dev0000055

PubMed Abstract | CrossRef Full Text | Google Scholar

Schmitt, S. A., Geldhof, G. J., Purpura, D. J., Duncan, R., and McClelland, M. M. (2017). Examining the relations between executive function, math, and literacy during the transition to kindergarten: A multi-analytic approach. J. Educ. Psychol. 109, 1120–1140. doi: 10.1037/edu0000193

CrossRef Full Text | Google Scholar

Schmitt, S. A., Korucu, I., Napoli, A. R., Bryant, L. M., and Purpura, D. J. (2018a). Using block play to enhance preschool children’s mathematics and executive functioning: A randomized controlled trial. Early Child. Res. Q. 44, 181–191. doi: 10.1016/j.ecresq.2018.04.006

CrossRef Full Text | Google Scholar

Schmitt, S. A., Lewis, K. M., Duncan, R. J., Korucu, I., and Napoli, A. R. (2018b). The effects of positive action on preschoolers’ social–emotional competence and health behaviors. Early Child. Educ. J. 46, 141–151. doi: 10.1080/10409289.2013.825565

PubMed Abstract | CrossRef Full Text | Google Scholar

Schmitt, S. A., McClelland, M. M., Tominey, S. L., and Acock, A. C. (2015). Strengthening school readiness for head start children: Evaluation of a self-regulation intervention. Early Child. Res. Q. 30, 20–31. doi: 10.1016/j.ecresq.2014.08.001

CrossRef Full Text | Google Scholar

Siegler, R. S. (1994). Cognitive variability: A key to understanding cognitive development. Curr. Dir. Psychol. Sci. 3, 1–5. doi: 10.1111/1467-8721.ep10769817

CrossRef Full Text | Google Scholar

Siegler, R. S. (2007). Cognitive variability. Dev. Sci. 10, 104–109. doi: 10.1111/j.1467-7687.2007.00571.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Sluss, D. J., and Stremmel, A. J. (2004). A sociocultural investigation of the effects of peer interaction on play. J. Res. Child. Educ. 18, 293–305. doi: 10.1080/02568540409595042

CrossRef Full Text | Google Scholar

Tominey, S. L., and McClelland, M. M. (2011). Red light, purple light: Findings from a randomized trial using circle time games to improve behavioral self-regulation in preschool. Early Educ. Dev. 22, 489–519. doi: 10.1080/10409289.2011.574258

CrossRef Full Text | Google Scholar

Wolfgang, C. H., Stannard, L. L., and Jones, I. (2001). Block play performance among preschoolers as a predictor of later school achievement in mathematics. J. Res. Child. Educ. 15, 173–180. doi: 10.1080/02568540109594958

CrossRef Full Text | Google Scholar

Zelazo, P. D. (2006). The dimensional change card sort (DCCS): A method of assessing executive function in children. Nat. Protoc. 1, 297–301.

Google Scholar