Social-Emotional Learning Amidst COVID-19 School Closures: Positive Findings from an Efficacy Study of Adventures Aboard the S.S. GRIN Program

Li, Linlin; Flynn, Kylie S.; DeRosier, Melissa E.; Weiser, Gary; Austin-King, Kelsey

doi:10.3389/feduc.2021.683142

ORIGINAL RESEARCH article

Front. Educ., 21 June 2021

Sec. Educational Psychology

Volume 6 - 2021 | https://doi.org/10.3389/feduc.2021.683142

This article is part of the Research Topic Rigorous and High-Quality Efficacy Studies of Educational Technology Interventions View all 8 articles

Social-Emotional Learning Amidst COVID-19 School Closures: Positive Findings from an Efficacy Study of Adventures Aboard the S.S. GRIN Program

Linlin Li¹^*

¹WestEd, Alameda, CA, United States
²3C Institute, Durham, NC, United States

School closures because of natural phenomena, such as COVID-19, have emphasized the importance of effective distance learning strategies when there are no in-person alternatives. Although infrequently mentioned in discussions of student performance, social-emotional skill building came to the forefront of conversations due to the isolation and stresses created by stay-at-home protocols. Our original research study describes the implementation of a game-based online social-emotional learning program during in-person learning and how we adapted implementation to distance learning due to COVID-19 based on schools’ infrastructure, preparations, and resource availability. In addition to the successful implementation, the results have indicated that the program was significantly and positively associated with gains in students’ social emotional skills.

1 Introduction

The COVID-19 school closures have underlined the importance of effective distance learning strategies when there are no in-person school alternatives. Since the spring of 2020, temporary school closures in over 180 countries have kept 1.6 billion students out of school (Azevedo, 2020; World Bank, 2020). The stressors associated with the pandemic and lack of access to schools (e.g., loss of routines, social isolation, economic hardships, lack of technology, no school meals) have increased the need for effective social and emotional learning (SEL) programs that can be delivered remotely. This paper discusses findings from an efficacy study funded by the Institute of Education Sciences that examined the impacts of a game-based online SEL training program designed to improve social and academic outcomes for elementary school students. This study began with in-person school implementation before COVID-19 and transitioned to distance learning during stay-at-home orders. In order to provide effective distance learning and ensure students’ engagement with the SEL platform, the study teachers, researchers, and program developers worked together to address students’ needs based on the schools’ infrastructure, preparations and resource availability.

Healthy social-emotional development is particularly important because it impacts the whole child. Students with good social-emotional skills have better academic performance and school behaviors (Durlak, et al., 2011; Payton et al., 2008), fewer behavior problems (Feinberg, et al., 2007), better relationships with peers and family (Chow, et al., 2013; Crawford & Manassis, 2011), and fewer mental health issues (Groeben, et al., 2011; Röll, et al., 2012). Students who have difficulty navigating the social developmental shifts of school are more likely to experience academic underachievement, behavior problems, and emotional difficulties (Kupersmidt & DeRosier, 2004; Najaka, et al., 2001; Parker et al., 2006). In addition, poor social skills place students at heightened risk for bullying, teasing, and social isolation (Asher, et al., 1996; Olweus, 1993; Solberg, et al., 2007). As students experience social failure, school becomes an aversive place which they may actively try to avoid (Gazelle & Druhen, 2009; Ladd, 2006). In fact, there is a direct relation between being bullied and teased at school and greater absenteeism, which in turn negatively impacts students’ academic performance (DeRosier, et al., 1994). If social-emotional challenges go unaddressed and peer problems become more chronic and severe, the likelihood of serious negative outcomes in adolescence, including academic failure (DeRosier et al., 1994; Fleming et al., 2005; Woodward & Fergusson, 2000) and dropout (French & Conrad, 2001), significantly increases.

Despite availability of proven SEL programs, many students who could benefit currently do not receive the help they need due to logistical barriers to implementation that limit the reach of traditional in-person SEL (Mueser & Bellack, 2007; Agron, et al., 2010;Center for Mental Health in Schools at UCLA, 2014). For SEL instruction to be effective, schools must implement the evidence-based program with fidelity (Durlak et al., 2011; Ringwalt, et al., 2009; Weist, 1997). Fidelity in SEL might be compromised for several reasons. SEL in schools is typically delivered by school support staff (e.g., school counselor), and there is often inadequate funding to cover their time. Competing priorities, lack of time from overloaded staff, inadequate tools to support ongoing implementation, insufficient training, and instructional drift over time can erode fidelity and sustainability (Agron et al., 2010). Student engagement can also be an issue, with providers commonly reporting that motivating children to engage in traditional interventions is among their most difficult tasks (Crenshaw, 2008). Finally, for many students and families, there remains a stigma associated with receiving mental health services, and an in-person intervention might dissuade some from seeking a treatment that could help. Taken together, these obstacles prevent many high need students from receiving effective, high quality SEL instruction and present a significant barrier to bringing effective SEL practices to scale. The stressors associated with the pandemic and lack of access to schools (e.g., loss of routines, social isolation, economic hardships, no school meals) have exacerbated these barriers and increased the need for effective SEL programs that can be easily and flexibly implemented.

Technology-based interventions provide the opportunity to reduce these barriers to treatment and effectively deliver evidence-based SEL content with fidelity (Thomas, 2014). Technology- and game- based programs can be delivered at a much lower cost and on a flexible schedule as time allows, increasing opportunities for exposure to the program and maximizing the number of students who can benefit. However, for technology- and game-based SEL to be effective and result in observable improvement in real world settings, the instructional content must be built on a solid theoretical and research foundation. The Adventures Aboard the S.S. GRIN program (henceforth Adventures) was specifically designed to translate the content and cognitive-behavioral strategies of an established, evidence-based in-person SEL intervention into a game-based environment. It is a comprehensive program that addresses all five social-emotional competency clusters identified by the Collaborative for Academic, Social, and Emotional Learning, which include self-awareness, self-management, social awareness, relationship skills, and responsible decision making (Sanchez et al., 2017).

Adventures has a very low burden on teachers because it is (a) short in duration (only nine 30–45-minute episodes), (b) does not require intensive coaching or professional development (simply a one-hour webinar), and (c) is supplemental, so does not require a major change in practice. The instructional topics included in Adventures are shown in Table 1. To maximize engagement for the target population, the instructional episodes are set within an appealing story narrative: the player is a new recruit on the sailing ship S.S.GRIN who joins the crew and travels around an island, interacting with a host of characters and engaging in social problem solving to address plot conflicts and save friendship. The social problems encountered in the game are true-to-life (e.g., entering group social situations, staying calm in emotionally charged situations, expressing emotions positively, cooperating and compromising with peers) to increase ease of skill transfer to real life. Lines of mystery and “cliff-hangers” are woven throughout the story to maintain engagement across episodes.

TABLE 1

Table 1. Episode Topics

As a practical, research-based, and theoretically sound technology-based approach to improving social skills in students, Adventures is intended to be implemented in typical third-, fourth-, and fifth-grade classes, providing nine weeks of instruction through interactive episodes. By utilizing a technology-based game-like format, the Adventures program overcomes the barriers associated with traditional SEL programs and capitalizes on the advantages of online delivery. In this way, the program can bring an effective SEL program to scale and make it available for many more students while reducing the burden on counselors and schools. As illustrated in the Theory of Change (Figure 1), this process can be stated in five steps. (1) Students use the true-to-life scenarios in Adventures to learn and practice new or uncomfortable social skills in a “safe” environment in which poor choices are corrected gently and without the social stigma that often accompanies in-person social mistakes. Within the program, students can practice these fledgling skills to mastery until they are comfortable using them. As they master the social skills they have struggled with in the past, their self-efficacy for social interactions will increase (DeRosier 2004; DeRosier & Marcus, 2005). (2) Using the newly mastered social skills will increase the number of positive behaviors and decrease the number of negative behaviors exhibited by the students (Beidel et al., 2005; Payton et al., 2008). (3) Better behavior will result in improved social interactions, fewer disruptive student behaviors, and increased student engagement as students are better able to focus on academic learning (Zins et al., 2004). (4) Less time spent addressing disruptive behaviors and maladaptive social interactions will result in increased instructional time, a reduction in disciplinary actions, and lower rates of absenteeism (Durlak et al., 2011; Bradshaw et al., 2010). (5) This increased instruction time and focus will result in an improved academic learning context, better grades and standardized test scores, and ultimately reduced dropout rates (Linares et al., 2005).

FIGURE 1

FIGURE 1. Logic model for the Adventures Aboard the S.S. GRIN intervention.

Guided by the logic model (Figure 1), this study attempts to address the following research questions, which were adapted as the study progressed in order to consider new circumstances emerging from the COVID-19 pandemic and resulting distance learning:

RQ1. Is the Adventures program treatment more effective than the control for improving students’ social-emotional skills?

RQ2. How was the Adventures program implemented during in-person learning and distance learning?

2 Materials and Methods

2.1 Study Design

The study used a multi-site cluster randomized experimental design, which randomly assigned third-grade classrooms to a treatment or control group. A total of 37 third-grade classrooms from 19 schools in three California public school districts participated in the study. The number of classrooms per school ranged from one to four. For schools with an even number of participating classrooms, we randomly assigned classrooms to one of two groups—a treatment group that used Adventures, or a control group that used the school’s business-as-usual SEL activities within each school. For schools with an odd number of participating teachers, we applied optimal Mahalanobis matching to pair schools, where we first calculated the Mahalanobis distance between all schools with an odd number of participating teachers based on four matching variables (percent of students who received free and reduced lunch program, percent of English Learners, percent of Latinx students, and percent of White students). We conducted 2000 random pairing configurations between schools and selected the pairing configuration with the lowest average Mahalanobis distance across pairs. We then randomly assigned teachers within each pair to the treatment or control group. Eighteen classrooms (n=395 students) were randomly assigned to treatment and 19 classrooms (n=428 students) were randomly assigned to control.

2.2 Procedures

This study began with in-person school implementation in December 2019. As part of the recruitment process the research team provided teachers with an informational letter describing the study activities, risks and benefits, confidentiality, and ability of participants to withdraw at any time. Teachers signed a consent form provided with the letter to agree to participate in the study. Teachers distributed informational letters to their students’ parents and guardians, describing the study activities and instructing parents/guardians to sign an attached opt-out form if they did not want their child’s data included in the research. All family communication was provided in English, Spanish, Chinese, and Vietnamese. WestEd’s IRB waived the requirement of informed consent, permitting the use of an opt-out process, to reduce the burden on families and teachers, given that most students from each class were eligible to participate.

To prepare treatment teachers to implement Adventures effectively and with fidelity, the Adventures development and research teams provided systematic training and supports to the teachers. The treatment training focused on 1) background information about the Adventures development and the use of Adventures; 2) teacher facilitation of post-episode discussions to help students understand and apply social-emotional skills in real life situation; 3) instructions for successfully implementing Adventures in the classroom, including strategies for behavior management and technology use; and 4) preparing teachers to promote transfer of students’ use of Adventures to other opportunities inside and outside of school. In addition, the research team also provided both treatment and control teachers instruction on completing the teacher rating scales and collecting student assessment data (see Measures for detailed descriptions of the data collection tools).

The research team encouraged treatment teachers to implement Adventures for nine weeks, one episode per week, estimating 30–45 minutes per episode. Two additional weeks were added to allow teachers the opportunity to make up episodes with students as necessary, creating an 11-week implementation period. To maintain communication between the research team and the participating teachers and provide prompt study implementation information, the team used weekly digital newsletters, weekly online teacher implementation logs, phone calls, texts, and emails. The weekly newsletter provided teachers a reminder of the assigned episode for the week, implementation tips, and study updates. The weekly log asked teachers to report their Adventures implementation format (e.g., whole class, small group, synchronous, asynchronous), session attendance, the content/episode covered that week, other intervention services students received, and whether teachers had questions or concerns about the implementation. The research team utilized a shared study email address to communicate with teachers, carefully monitoring incoming messages and promptly addressing teachers’ questions. Via the shared email, in addition to phone and text, the research team coordinated with teachers to understand the factors impacting their implementation of the program and to provide support as needed for classes that were falling behind schedule. In this way, the research team was able to maintain frequent and up-to-date communication with teachers, as well as collect information on schools’ instructional priorities, technology capacities to support students’ access to distance-learning, and the feasibility of implementing Adventures in distance-learning when schools closed due to COVID-19. Most of the treatment classrooms were able to finish about six out of nine episodes before school closures in early March. With teachers’ and families’ demand and willingness to continue Adventures, the study was continued in distance-learning environments through May 2020.

Prior to school closures, students completed online measures of social-emotional skills under the supervision of their teachers before the implementation of Adventures. Teachers completed additional behavioral rating scales on a subsample of sixteen students per class— eight had the lowest scores on an SEL screener (see Devereux Student Strengths Assessment-Mini in Measures section below) and the other eight were randomly selected. After implementation concluded, students completed the same online social-emotional measures administered at the beginning of the study and teachers filled out the same behavior rating scales on the subsample of sixteen students. All post-study measures were collected after the transition to distance learning, with teachers supervising most students as they completed the social-emotional measures from home. The research team worked diligently to assist participating teachers in implementing the program and collecting needed data throughout distance learning. Teachers and researchers all made a tremendous effort to ensure a smooth transition from in-school implementation to distance learning.

2.3 Participants

The original study sample consisted of 823 third-grade students from 37 classrooms in California. Although the sudden transition from in-school instruction to distance learning instruction due to COVID-19 created challenges for implementing Adventures, 17 out of 18 treatment classrooms were able to continue the Adventures program in distance learning environments. Eighteen out of 19 control teachers were able to remain in the study and provide valuable inputs on their “business-as-usual” remote instruction experiences and practices.

Five hundred forty-three students (242 treatment students vs. 301 control students) completed at least one direct SEL outcome measure. Teachers provided SEL ratings for 480 students (228 treatment students vs. 252 control students). Given the circumstances of unexpectedly and rapidly transitioning to remote learning, however, it is not surprising that student completion of the post-assessment was low. Analysis of the student demographic data indicated that more than 50% of the final analytic sample were qualified for Free/Reduced-Price Lunch program and 46.8% are Latinx. There were no statistical differences between treatment and control on their ethnicity, Free/Reduced-Price Lunch status, English language learner status, and gender. Tables 2 and 3 provide the demographic information for the full impact sample and subsample respectively.

TABLE 2

Table 2. Student demographic information for the full impact sample by condition

TABLE 3

Table 3. Student demographic information for teacher-rated impact sample by condition

To examine how data attrition might have affected the baseline equivalence between the treatment and control groups, the research team conducted a baseline equivalence test for each outcome (Table 4). There were no statistical differences between treatment and control at baseline for both the full sample and the subsample.

TABLE 4

Table 4. Baseline differences in the pre-assessment scores

2.4 Measures

2.4.1 Screening Measure

2.4.1.1 Devereux Student Strengths Assessment-Mini (DESSA-Mini)

The DESSA-Mini is an 8- item behavior rating scale used to screen for and monitor progress in the acquisition of social-emotional competencies of K-8 students. The DESSA-Mini provides one score, the Social- Emotional Total (SET), which summarizes a student’s overall social-emotional competence. The measure is nationally standardized and has excellent psychometric properties, with internal reliability of .92, test-retest reliability of .94, and inter-rater reliability of .77 (Naglieri et al., 2011). The validity study indicates the DESSA-Mini can be used with confidence as a screener for social-emotional competence. DESSA-Mini SET scores are strongly correlated with the Social-Emotional Composite scores on the full DESSA regardless of race or ethnicity. Both assessments identify the same children as being in need of social-emotional intervention. In addition, the DESSA-Mini SET scores clearly differentiate groups of children with and without known social-emotional problems (Naglieri et al., 2011). Students whose SET score is 40 or below, which corresponds to one or more standard deviations from the established norm, are identified as at risk for developing social-emotional problems and therefore in need of intervention. Before implementation began, each teacher was asked to complete this measure for each student in their third-grade classroom.

2.4.2 Teacher Ratings of Student SEL Outcomes

2.4.2.1 Behavioral and Emotional Rating Scale Second Edition (BERS-2)

The BERS-2 is used to measure students’ social-emotional outcomes. The BERS-2 (52 items) is a standardized and norm-referenced assessment that documents emotional and behavioral strengths of children between the ages of five and 18 years within a school setting. The BERS-2 consists of six subscales: interpersonal strength, involvement with family, intrapersonal strength, school functioning, affective strength, and career strength. For the purpose of this study, the research team used the subscale of interpersonal strength which includes 15 items, with Cronbach's alpha coefficient of .96 (Epstein, 2004; Epstein et al., 2002b). In a large national sample, the BERS-2 was shown to differentiate between students with emotional-behavioral disorders and those without, which indicated good criterion validity (Epstein et al., 2002b). Several studies have been conducted to assess the convergent validity of the BERS-2. The BERS-2 consistently demonstrated moderate to high correlations with competence-oriented scales on the Social Skills Rating Scale and moderate to high negative correlations with deficit-oriented scales across different age ranges (Epstein et al., 2002a; Epstein et al., 2004).

2.4.2.2 Social Skills and Behavioral Index (SSBI)

The SSBI is a researcher-developed assessment (DeRosier, 2011). The full rating scale measure includes 75 rating scale items assessing six social skills scales (eight communication items, seven cooperation items, six empathy items, six social initiation items, six impulse control items, and six emotion regulation items), two behavioral scales (26 items covering a variety of Internalizing and Externalizing behavior problems), one social acceptance scale, and one academic performance scale. Data collection can include one or more scales and subscales, as desired. For the study, we used the communication skills scale (eight items). Prior studies have demonstrated acceptable levels of internal consistency for each scale, with a Cronbach's alpha estimation for the communication skills scale of 0.914.

2.4.3 Student Direct Assessments on SEL Learning

2.4.3.1 SELweb

SELweb is a performance-based assessment of social-emotional skills (McKown et al., 2016) that examines students’ skill across 12 scenarios. Each scenario provides an opportunity for students to identify a pro-social goal when faced with an ambiguous situation and then determine a solution that is most likely to lead to a positive consequence. As a distal measure, SELweb provides two scores, one for goals and another for solutions. Analyses supported SELweb’s criterion-related, convergent, and discriminant validity. SELweb’s score reliabilities averaged 0.72 and 0.78, respectively for the goal and solution subscales, and 0.84 overall.

2.4.3.2 Zoo U

The Zoo U is a game platform created by the Adventures’ developer that assesses social-emotional skills in children and is proximally aligned to the intervention. Users interact with characters in a school-like setting through six scenarios. As users play through the social situations, the platform collects performance-based data pertaining to communication, cooperation, empathy, emotion regulation, impulse control, and social initiation. Prior studies with upper elementary aged students have shown that Zoo U ratings significantly correspond with independent teacher ratings of social skills (DeRosier et al., 2012) and that the assessment tool demonstrated high internal consistency and reliability as well as criterion validity in the prediction of independently assessed school-based outcomes, including externalizing and internalizing behavior problems (DeRosier & Thomas, 2018).

2.4.4 Observations and Interviews

2.4.4.1 Site Visits

Site visits were conducted before school closures. Ten out of 17 treatment classrooms from eight schools were observed. The purpose of the site visits was to understand student behaviors, affect, and engagement when using Adventures. The Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) is a method of collecting data to measure student engagement. BROMP was designed for observing students while they use educational technology and is well-suited to that purpose. When using BROMP researchers pre-determine an order in which they will observe students, then move sequentially through that order, recording a behavior and an affect for each student. Researchers observe students sequentially for the duration of the activity, cycling through the ordered list of students as many times as the session allows. Observations were recorded using a mobile device-compatible application that was designed specifically for BROMP observations. Observations captured behavior in three categories (on task, on task conversation, and off task), and affect in four categories (concentrating, bored, confused, and frustrated).

All researchers who collected BROMP data for this study were certified, with interrater reliability (Kappa) scores ranging from 0.82 to 0.90. Data represent all three districts in the study. The average observed class contained 23 students, with the class size ranging from 15 to 27. The average length of the observed sessions was 29.5 minutes; the shortest session ran for 10.7 minutes, and the longest lasted 49 minutes. Researchers recorded 928 observation instances across the ten sessions.

2.4.4.2 Teacher Interviews and Surveys

Teacher interviews and surveys were conducted at the end of the study. Treatment teacher interviews focused on: (a) teachers’ use of Adventures; (b) student engagement and learning; (c) quality and content of Adventures; (d) feedback on implementation success and challenges, including the transition to distance learning, and (e) recommendations for future implementation. Control teacher surveys focused on their social-emotional instructions in school and during distance learning.

3 Results

3.1 Effectiveness of the Adventures Program on Students’ Social-Emotional Skills (RQ1)

To address whether the Adventures program treatment is more effective than the control for improving students’ social-emotional skills (RQ 1), the research team used a set of two-level hierarchical linear models (fixed block effects and fixed treatment effect models), where the dependent variable in each model is a separate SEL outcome. In each model, the effect of treatment on an outcome measure is estimated at the classroom level, controlling for student pretests, baseline student covariates (i.e., gender, ethnicity, and eligibility for Free and/or Reduced-Price lunch programs). The blocks were treated as the fixed effect in the models.

Level-1: Student Level

Y_{i j} = β_{0 j} + β_{1 j} (Y_{i j}^{*}) + \sum_{m = 1}^{M} β_{2. m} X_{m i j} + ε_{i j}

Level-2: Cluster (School) Level

β_{0 j} = γ_{00} + γ_{01} (T_{j}) + \sum_{p = 1}^{P - 1} γ_{02. p} B l o c k_{p j} + μ_{0 j}

β_{1 j} = γ_{10}

β_{2. m j} = γ_{2. m 0}

Where, $Y_{i j}$ is the outcome for the i^th student in the j^th class; $β_{0 j}$ is the intercept for class j; $β_{1 j}$ is the effect of pretest in class j; $Y_{i j}^{*}$ is a pre-test measure for the i^th student in the j^th class; $β_{2. m j}$ is the effects of student covariates in class j; $X_{m i j}$ is the m^th of $Μ$ additional covariates for student i in class j. $ε_{i j}$ is a residual error term for student i in class j; $γ_{00}$ is the mean intercept; $γ_{01}$ is the treatment effect; $T_{j}$ represents whether class j is assigned to treatment or control; $μ_{0 j}$ is the random intercept term (deviation of block j’s mean from the grand mean, conditional on covariates; assumed to be normally distributed with mean 0 and variance $τ_{00}^{2}$ ); $γ_{10}$ is mean effect of pretest; $γ_{2. m 0}$ is mean effect of student covariate m; $B l o c k_{p j}$ indicated whether the class j was assigned to the treatment or control condition within the block p; and $γ_{02. p}$ is the effect of block p.

The stepwise multiple hypothesis testing procedure was applied to account for conducting multiple comparisons (Benjamini & Hochberg, 1995). This procedure involves rank ordering outcomes for each student by from smallest to largest p-values, multiplying a critical p-value of 0.05 by j/N, where N is the number of student outcome variables under one domain (N=2 for the full sample and N=4 for teacher-rated subsample) and j represents the order of the test. The procedure involves only rejecting null hypotheses in which the p-value is less than the largest p-value found that is lesser than its corresponding adjusted critical p-value.

The fixed effects from the impact analyses and effect sizes for the treatment variable are presented in Table 5, along with adjusted means and unadjusted standard deviations for treatment and control on each measure by the full- and sub- samples. Overall, the Adventures program was significantly and positively associated with gains in students’ social-emotional skills. The effect sizes of the treatment impact as measured by different outcome measures ranged from .29 to .63 (based on Hedges’ g), which is medium to large for educational interventions for a sample of this size (Kraft, 2020).

TABLE 5

Table 5. The effect of Adventures on student social-emotional skills

For the full sample, the estimated average treatment effect was statistically different between treatment and control as measured by Zoo U (0.33 vs. -0.24; effect size=0.63; p<.001). The treatment group scored 3.87 points higher than control group on SELweb overall scores (45.66 vs. 41.79; effect size=0.33; p<.001), 1.54 points higher on SEL Goal subscores (18.22 vs. 16.68; effect size=0.31; p<.001), and 2.41 points higher on SEL Solution subscores (27.48 vs. 25.08; effect size=0.32; p<.001).

For the subsample of teacher-rated group, the estimated average treatment effect was statistically different between treatment and control as measured by student direct assessments and teacher reported ratings. The treatment students surpassed control students on Zoo U assessment by 0.58 points (0.33 vs. -0.25; effect size =0.61; p<.001). The treatment group scored 4.16 points higher than control group on SELweb overall scores (45.32 vs. 41.16; effect size =0.37; p<.001), 1.64 points higher on SEL Goal subscores (18.08 vs. 16.44; effect size =0.34; p<.001), and 2.56 points higher on SEL Solution subscores (27.26 vs. 24.69; effect size =0.36; p<.001). The impact on student SEL skills as measured by teacher ratings showed similar results. The treatment group students received higher SEL scores compared to control. The treatment group students outscored the control students on SSBI (33.13 vs. 31.34; effect size=0.29; p<.05) and BERS (16.17 vs. 14.87; effect size=0.39; p<.01).

3.2 In-School and Distance Learning Implementation (RQ2)

3.2.1 In-School Implementation

Before school closures, about 92.9% of the treatment students were able to finish six out of nine episodes. From observations and interviews, researchers observed and teachers reported a high level of student engagement with the game during the in-school implementation. Teachers provided students with an average of about 45 minutes to complete each episode, offering extra time to students who were unable to finish. Teachers often implemented Adventures at the beginning of the week and allowed for time later in the week for students to finish up or make-ups. In interviews, teachers reported preferring to use the program as a whole class, as they felt it was a great activity for all students to learn the SEL skills together and acquire a shared understanding.

Episode discussions often occurred later in the week to make sure most students had finished playing the assigned episode. Teachers reported that episode discussions were a good opportunity for the class to discuss the SEL topics together and that students often engaged in a deep level of reflection.

"We would have a meaningful long conversation about those episodes, that I would have to cut it short sometimes."

"One of the post discussions that we were having, one of my students who, I think he's labeled [with an intellectual disability] on his IEP, he was so into the conversation. He told me how he passed this lesson, and what he needed to accomplish. It was really good. After that, I was blown away, and I was like, 'I have full faith in this program. This kid understood.’"

“I've noticed a few of my students who were shy typically, who don't really share out that often, they were the ones that were more outspoken about the game.”

BROMP observation data analysis results also indicated high engagement. In particular, 98.5% of the total recorded observation instances (928 observation instances total) captured on task behaviors; researchers observed only 13 instances (1.4%) of off task behavior. Researchers distinguished between students going off task within the platform (e.g., exploiting a software quirk to navigate the avatar up a wall, or some other action within Adventures that does not align to the learning task) and students going off task such that they were not using Adventures productively (e.g., talking to a neighbor). Only one instance of this former type of “in game” off task behavior occurred. These data strongly indicated that Adventures gameplay was compelling and did a good job of holding students’ attention, and that Adventures did not offer many opportunities for students to deviate from its intended use—if a student’s attention is directed towards the program, they are likely on task. The affect data further support these conclusions. A large percentage of students displayed a concentrating affect (91.3%), with the next most commonly displayed state being delight (4.2%). Students exhibited occasional but highly infrequent states of frustration (1.4%), boredom (1.29%), confusion (1.19%), and disappointment (0.32%). Overall, students were observed as being engaged and occasionally delighted with the game, with rare moments of negatively valanced emotion.

3.2.2 Distance Learning Implementation

As previously noted, because of the COVID-19 pandemic and the shelter-in-place order in California, all of the schools transitioned from in school instruction to distance-learning starting in mid-March 2020. To determine the viability of the research study continuing, our research team quickly reached out to teachers via emails, phone calls and online video meetings. Teachers in the treatment and control groups reported that they used both synchronous and asynchronous instructional strategies during distance learning to address the varying needs of family schedules and technology accessibilities. Most teachers report using a variety of platforms to communicate with students during closures. Teachers in all three districts reported using Class Dojo. Other platforms used include Google Classroom, Seesaw, Remind, Zoom, Clever, Google Meet, and ClassTag. Teachers also created paper packets for students to pick up and provided instructional support via online platforms. Based on these reports, along with teachers’ desire to continue, the study activities resumed after a four-week pause.

Although each classroom had students who did not have devices or internet connections and it was hard for teachers to support the students due to inconsistency in devices at home, family comfort with technology, and other factors, 74.4% of the students were able to engage in all nine episodes during distance learning. Across classrooms, teachers remarked that students were excited to be able to continue using Adventures at home. Many enjoyed the chance to work at their own pace.

“I've opened it up where they [students] could work at their own pace at home. … I put it on my grid on the SEL portion for them to work on it. … the majority of them has worked on it at least once at home, but I think there's still two that have not touched it since March.”

“We were allowed to assign two-hour worth of work. I just said, “30 minutes, here's an episode.” Like I said, we left off on [Episode] Seven. The week before we started distance learning, I used it as a check-in. “Remember, those of you who didn't finish Seven, please tell me. Go on Google Classroom, type ‘done’ when you're done.” Then from there, [Episodes] Eight and Nine, I just did it as a weekly assignment.”

“It did allow those who wanted to just keep going. That was one part and it did give them another…“had-to-do,” a “must-do,” but it was a fun “must-do.” It changed up the rhythm of things, because I try to give them fun stuff to do, critical stuff and things like that.”

“They were very happy that they got to finish it. I know they were bummed because they only had three left and they wanted to know how it ended. That was really good.”

A majority of teachers (all but three) were not able to conduct the post-episode discussions during distance-learning due to the complexity of adjusting their class routine to accommodate students’ needs on flexible scheduling and technology sharing among family members. Those who were able to hold the post-episode discussions saw this as a great way for the class to stay connected on Adventures and continue conversations regarding SEL.

“I did it exactly the way I did it at school. I had a little cheat sheet booklet. I had the posters and I would just hold the poster up blah, blah, blah. Then I would go through those questions and I would call on kids because I'd mute them all. Then I would call on one at a time to answer the questions. If they had a different answer, I'd call them. Basically exactly the same I did it in class.”

4 Limitations and Directions for Future Research

Despite the success of transitioning the Adventures program to a distance learning context, there were several limitations imposed by the pandemic and sudden school closures. First, while the original sample of students implementing the program pre-COVID includes 823 students across 37 classrooms, after the school closures, the sample size decreased to 543 students spread among 35 classrooms (66% of the students remained in the study). Not only did two classrooms decline continuation of the program implementation, but within each remaining classroom, there was variation in distance learning participation. This is not surprising, given the circumstances. The baseline equivalence test finding, coupled with teachers’ reports during interviews, suggest that students dropped out of the study after COVID-19 due to the instructional changes (from in-school to distance learning) and accessibility of devices and internet, rather than features related to the program itself. Second, the study focuses on the overall impact of the program on students without considering the variations on schools’ infrastructure and technology capacity to support student learning remotely; teachers’ experience, tools, and techniques to meet student needs; and students’ needs for devices, internet, and in-person supervision by adults. Further analyses are needed to address whether the program is more effective than business-as-usual for developing social-emotional skills for subgroups of students (e.g., students from poverty backgrounds), and to what extent dosage of the Adventures program impacts the benefits students receive from the program. The authors are conducting a second cohort of the study and will address these limitations through additional analyses when the study is sufficiently powered with two cohorts of data.

As an Intelligent Social Tutoring Software, the Adventures program tracks children’s interactions and calculates performance indices for each social problem-solving task and for overall achievement. When students navigate Adventures, the software captures in-game behaviors and generate telemetry data. Future studies can use the Adventures telemetry data to explore whether the Adventures implementation dosages necessary for achieving the proficient level in gameplay varies for different sub-groups of students and whether customized exposures for sub-groups can yield flexible implementation strategies that still benefit all students.

The stressors associated with the pandemic and lack of access to schools (e.g., loss of routines, social isolation, economic hardships, no school meals) have increased the need for effective SEL programs that can be easily and flexibly implemented (Styck et al., 2020). Our findings underscore the potential for game-based online SEL programs to improve outcomes for students. Such findings raise additional questions that drive ongoing research in the field. These research questions, for instance, include whether findings replicate in novel contexts or among select populations, whether certain features of the product or its implementation were most critical to the observed efficacy of treatment, and whether the impact on students can expand to include academic performance. Some of these questions are currently under research by the authors as part of a second cohort of participants who joined the study during the 2020-21 school year as the effects of COVID-19 seemed to wane (Meckler & George, 2021). With both cohorts, we will be able to consider both ends of the pandemic, potentially capture academic performance metrics, and delve deeply into the gameplay data to learn more about the Adventures treatment and potential mediators to its impact.

5 Discussion

This study is important for advancing the research on the impact of COVID-19 on students for four reasons. First, it addresses social-emotional learning, which is even more critical given the increased stressors on students related to the pandemic. Second, the online, digital SEL platform allowed teachers and students the flexibility to finish the project activities remotely, making it a feasible option during ongoing distance learning. Third, even with the transition to remote implementation, the findings indicate significant and meaningful impacts on student outcomes. Fourth, despite the encouraging results, there were still a number of students (more than 20% of the overall sample) who were unable to complete the project activities due to a lack of devices, internet, or in-person supervision by their teacher, raising an important equity issue related to remote learning. One teacher reflected, “Basically I had no control over whether they did [Adventures] or not. It was all based on if they wanted to do it, if their parents wanted to do that,” voicing the challenge of remotely motivating students and ensuring they complete their work shared by many teachers.

The advent of affordable computer hardware, high speed Internet connectivity, and an expanding array of accessible technology platforms have made digital delivery of SEL programs increasingly feasible and attractive for schools and families. While the pandemic has highlighted inequities in Internet access for families and communities, schools can effectively address this access gap. A 2020 report by the nonprofit advocacy organization EducationSuperHighway found that 99% of U.S. schools have fiber-optic Internet connections that meet the FCC’s per student minimum bandwidth goal, enabling digital learning at school for the vast majority of U.S. students. However, actual adoption and use of game-based and online technologies to engage students in SEL instruction at school has historically lagged behind availability of such innovations and access to Internet. Skepticism about the value of and need for more novel approaches to SEL instruction continues, despite an ever-growing research literature supporting the efficacy, acceptability, and feasibility of digital approaches to SEL.

However, in the wake of the COVID-19 pandemic, the need and demand for digital supports for SEL have grown significantly. For example, Centervention, the publisher of Adventures, saw a 300% increase in usage of Adventures by schools from Fall 2019 to Fall 2020. With school closures and distance learning, use of digital platforms for the full array of instructional topics is now necessary, rather than secondary. Educators must look for new ways to engage students in SEL and assess their progress. The learning loss associated with the pandemic is stark. Students on average could lose five to nine months of learning by the end of June 2021. Students of color could be six to 12 months behind, compared with four to eight months for white students (Dorn et al., 2020). Further, early investigations of the pandemic’s impact on youth show they are experiencing heightened emotional distress (Jiao et al., 2020; Roccella, 2020). Social distancing has also impacted youth’s access to and support from peers, leading to increased social isolation and loneliness (Crawley et al., 2020). Specifically, students are being exposed to increased psychosocial adversity and are in even greater need of social-emotional supports. Unfortunately, the closure of schools leaves students without the usual supports and impedes schools’ ability to offer SEL instruction for their students. The combination of increased psychosocial stress and decreased access to psychosocial supports places students at particularly heightened risk for developing social-emotional problems and concomitant academic and behavioral delays. Scalable digital approaches to evidence-based SEL can meaningfully address the social and emotional needs of many students (Kazdin & Rabbitt, 2013). As demonstrated by the findings of this research, Adventures has offered part of a solution to this critical need.

6 Conclusion

While all participating classes in the study faced challenges, there were variations in the kinds of challenges experienced and how teachers overcame hurdles. Once converted to a distance learning program, each implementation was unique. This study highlights flexibility as a critical feature of game-based online SEL program implementation in a distance learning environment. Due to its ease of use and low burden, teachers were able to implement Adventures in in-school and distance-learning environments with fidelity. The results indicated that the Adventures program was positively and significantly associated with gains in students’ interpersonal strength as measured by BERS-2 (effect size=0.39), communication skills as measured by SSBI (effect size=0.29), pro-social goals and solutions to achieve positive consequences as measured by SELweb (effect size >=0.33), and social skills related to communication, cooperation, empathy, emotion regulation, impulse control, and social initiation as measured by Zoo U (effect size >=0.61). In addition, treatment teachers reported that students were engaged in and excited to continue using Adventures at home. They viewed Adventures as a great way for students to stay connected to their SEL.

While those in the education field seek answers on how to deliver instruction when students are required to stay home, future research work can focus on understanding which of the available remote learning strategies are most effective—with or without the internet, web-enabled devices, and comprehensive educational supports. Intelligent tutoring systems, such as Adventures, provide the potential to address student needs in distance learning. Developers of intelligent tutoring systems can further leverage play and narrative episodes in their designs and make interactives that are engaging and meaningful to students by building socio-technical structures that engage users, allow for a continual growth of individuals within the communities and cultures in which they are nested, and encourage active learner and child-centered learning (Barab et al., 2005).

Data Availability Statement

The dataset for this article is not publicly available because the article focuses on the first cohort of the study and the second cohort of the study is yet completed. The data supporting the conclusions of this article will be made available after both cohorts of data are collected and analyzed. Requests to access the datasets should be directed to bGxpQHdlc3RlZC5vcmc=.

Ethics Statement

The studies involving human participants were reviewed and approved by the WestEds Internal Review Board. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author Contributions

Study conception and design, LL, KF, MD. Acquisition of data, LL, KF, GW, KA-K. Analysis of Data, LL, GW, KA-K. Interpretation of data, LL, KF, GW. Drafting of manuscript, LL, KF, MD, GW, KA-K. All authors contributed to the article and approved the submitted version.

Funding

This study has been funded by a U.S. Department of Education Institute of Education Sciences Education Research Grant (grant number R305A180224).

Conflict of Interest

Authors LL, KF, GW, and KA-K were employed by the research agency WestEd. Author MD was employed by the company 3C Institute, the developer of the Adventures aboard the S.S.GRIN program. All study design activities, recruitment and random assignment, data collection, analysis, and interpretation of the impact are independently conducted by WestEd. The authors declare that the research was conducted in the absence of any commercial or financial relationship. The independence of the study ensures the objectivity and prevents the appearance of a conflict of interest.

References

Agron, P., Berends, V., Ellis, K., and Gonzalez, M. (2010). School Wellness Policies: Perceptions, Barriers, and Needs Among School Leaders and Wellness Advocates*. Journal of School Health 80 (11), 527–535. doi:10.1111/j.1746-1561.2010.00538.x