A mixed methods evaluation of the usability of Feel Your Best Self

Feel Your Best Self (FYBS) is an online toolkit developed to facilitate teaching emotion-focused coping skills to elementary-aged children. This study presents findings from a mixed methods evaluation of FYBS usability. Survey participants (n = 29) and interview participants (n = 12) shared their perceptions of FYBS, including strengths and areas for growth along with adaptations made relevant to their unique contexts. Results indicated that participants found FYBS to be highly usable, and that both children and implementers responded well to FYBS. FYBS was easy to use, and implementers were able to personalize materials and indicated a desire for additional features to enhance usability. Findings provide considerations for the iteration of FYBS materials along with directions for features to explore in future research.

1 Introduction

The need for comprehensive mental health supports across a continuum of promotion to intervention continues to be paramount for children and adolescents. From 2016 to 2019, the rates of US emergency department visits for mental health concerns increased 25% for children and adolescents (Agency for Healthcare Research and Quality, 2022). In 2019, UNICEF and the World Health Organization (WHO) began organizing efforts to tackle growing rates of mental disorders worldwide among this population (UNICEF, 2019). Then, in 2020, the COVID-19 pandemic exacerbated rates of child and adolescent mental health challenges across the globe (Hossain et al., 2022). In 2021, the American Academy of Pediatrics, the American Academy of Child and Adolescent Psychiatry, and the Children’s Hospital Association issued a declaration of national emergency in child and adolescent mental health which reiterated the need to implement mental health care and behavioral services, specifically in schools (American Academy of Pediatrics, American Academy of Child and Adolescent Psychiatry, Children’s Hospital Association, 2021). This continues to remain true worldwide; for example, in the United Kingdom, just under 500,000 children and adolescents under age 18 are awaiting treatment for mental and behavioral health concerns following referrals from their general practitioner (Thomas, 2024).

One direction in a comprehensive mental health solution is social–emotional learning (SEL), which can be provided universally to all children as a promotion and prevention strategy. Although there are many frameworks and conceptualizations of SEL, it can be broadly defined as “..the process through which individuals learn and apply a set of social, emotional, and related nonacademic skills, attitudes, behaviors, and values that help direct their thoughts, feelings, and actions in ways that enable them to succeed in school, work, and life,” (p. 45, Brush et al., 2022). SEL domains across conceptualizations include cognitive (e.g., skills related to goal setting and achievement), emotion (e.g., skills related to emotion recognition, expression, and control), social (e.g., skills related to engaging and interacting with others), values (e.g., skills and traits related to prosocial, community-minded behavior), perspectives (e.g., embodying a positive view of self and the world), and identity (e.g., understanding and skills related to perception of one’s self and ability; Brush et al., 2022).

Studies on the efficacy of SEL programming have yielded promising results (Durlak et al., 2011; Cipriano et al., 2023). A review of 12 meta-analyses, for example, found that SEL consistently positively impacts children, increasing SEL skills and prosocial behaviors and decreasing conduct problems and emotional distress (Durlak et al., 2011). In their more recent meta-analysis, Cipriano et al. (2023) reviewed 424 studies from 53 countries involving over half a million children, finding that compared to control groups, children who participate in school-based SEL interventions experience significant improvements in emotional skills, attitudes, behaviors, peer relationships and academic achievement, along with improved school climate.

Many SEL programs are available for use universally in schools, such as class-wide. For example, the Collaborative for Academic, Social, and Emotional Learning (CASEL) provides a “Program Guide” where school personnel can determine priorities for an SEL curriculum and filter through to identify which program they may want to implement. Of the 85 suggested programs, 84 require staff training for proper implementation (Collaborative for Academic, Social, and Emotional Learning, 2013). Although training improves fidelity of program implementation (Sanetti and Collier-Meek, 2019), however, training sessions often require substantial resources (e.g., staff time, financial investment, travel). This can be particularly problematic in schools as reports of staff burnout and attrition continue to climb (Agyapong et al., 2022). Thus, it is important that universal SEL options be available that require low resource demand and can offer use beyond school settings to reduce the burden on schools to fully deliver and facilitate generalization of skills. One direction to accomplishing this dual focus is to attend to the “active ingredients” of effective interventions, thereby reducing the need to implement a comprehensive curriculum (Iovino et al., 2021). Released in 2022, Feel Your Best Self (FYBS) was developed with this goal, offering a simple solution to bolster learning of important emotion skills.

Within the emotion domain of SEL, specific skills include emotion knowledge and expression, emotion and behavior regulation, and empathy and perspective-taking (Jones et al., 2021). Emotion knowledge and expression includes recognizing, understanding, and labeling emotions, and expressing emotions in accordance with a given situation. Emotion and behavior regulation involves controlling the intensity and duration of emotional responses and ability to control or modify behavior to fit a social context. Finally, empathy and perspective-taking includes understanding the experiences and viewpoints of others. Across a review of the most common universal SEL programs targeting elementary students, only 21% of programs explicitly addressed these emotion skills in the majority of lessons (Jones et al., 2021).

FYBS is an online toolkit designed to teach 12 emotion-coping strategies that enable children to take a situation and determine an appropriate balance of positive and negative emotion. In doing so, FYBS facilitates what has been referred to as flexible emotion regulation (Kobylińska and Kusev, 2019). Emotion regulation is a mechanism that helps individuals cope with demands by exerting control over their experiences and expressions of emotion. Flexible emotion regulation involves both having a range of strategies available that can be used to regulate emotions and ability to select and use an appropriate strategy based on environmental context and personal characteristics.

FYBS draws on several sources of evidence. First, as summarized by Iovino et al. (2021), the introductory lesson and 12 emotion-focused coping strategies (e.g., deep breathing, cognitive restructuring) are well-established by research on self-awareness, self-regulation, social relationships, and emotional well-being. Developers mapped the strategies, guided by Gross’s seminal process model of emotion regulation (PMER; Gross, 2014). This model proposes that strategies can be used at different points in the emotion cue-to-response continuum, with some strategies used proactively and others used reactively. Based on this mapping, three categories of strategies were developed that function at different points in the PMER. Those categories include Connect with Others, Catch Your Feelings, and Calm Your Self. Connect with Others strategies function at the point of situation selection or modification and are about social relationships, and include offering or seeking connection. Catch Your Feelings strategies support self-awareness of one’s emotional state through refocusing attention (i.e., attentional deployment) or shifting thoughts (i.e., cognitive change). Finally, Calm Your Self strategies focus on self-soothing skills that settle the body (i.e., response modulation) or refocus attention (i.e., attentional deployment). The 12 individual strategies are arranged within these categories.

The FYBS toolkit includes multiple components that offer flexibility for use across multi-tiered systems of support: the introductory lesson and Feelings Forecast, short strategy videos with puppet friends, strategy step cards, discussion tip sheets, reflection sheets, and puppet-making options. Facilitator guidance is available for using the various components. The introduction and strategies are presented in short lessons to increase options and opportunities for implementation. The use of puppets as an instructional technique for preschool (e.g., ages 3–5) and elementary aged children (e.g., ages 5–12) has also shown promising results for increasing student motivation, engagement, and discourse (Kröger and Nupponen, 2019). Additional description of FYBS can be found in Koslouski et al. (2024) or directly at the website.¹

Users can choose to teach any number of the 13 FYBS lessons. It is recommended that implementers start with the introductory lesson and Feelings Forecast as this lesson introduces children to the puppet characters featured in the videos and language used throughout materials to discuss feelings. However, users can then choose to teach remaining strategies in any order. Because FYBS offers opportunity for flexible implementation, it is unclear how current users haven engaged with the materials to teach the FYBS strategies—and therefore, whether they perceive the toolkit to be usable as the developers intended.

Industry standards have highlighted the importance of usability as an outcome of quality goods, services, and products. Usability refers to the extent to which a good, service, or product “…can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction…” in their context (International Organization of Standardization, 2018). Effectiveness refers to the extent to which users can achieve intended outcomes with accuracy and completeness. Efficiency refers to resources such as time, effort, financial costs, and materials that are used to achieve intended outcomes. Satisfaction refers to whether the resulting “physical, cognitive, and emotional responses” experienced by the user have met their needs or expectations (International Organization of Standardization, 2018). In schools, an essential component of generalized use of tools across settings and over time is usability (Braun and Clarke, 2021).

With respect to FYBS, the context of use can be schools, homes, clinical practices, community settings, and more. As previously noted, the program (i.e., product) was purposefully developed with the intention of being effective, efficient, and satisfying for implementers across these contexts. Although individuals who work in these different contexts participated in the development of the FYBS toolkit materials, it is unknown whether the intended outcome of usability is being achieved. As such, an important step in the continued development and testing of program materials is to understand whether current users across contexts find FYBS to be usable, thereby exploring perceptions of effectiveness, efficiency, and satisfaction.

The purpose of this study was to evaluate the usability of FYBS, and to understand how implementers have selected and adapted various materials. Our research questions included: (1) Do users of the FYBS toolkit perceive it to be usable? and (2) How have users adapted the FYBS toolkit to enhance their implementation? We hypothesized that findings would support high perceptions of usability, and that users would note adaptations to fit for their context.

2 Materials and methods

We used a convergent mixed methods design (Fetters et al., 2013; Moseholm and Fetters, 2017), which allowed us to collect quantitative survey data and qualitative interview data concurrently. Qualitative and quantitative data were first analyzed separately and then results combined for overall interpretation of usability.

2.1 Recruitment and participants

After obtaining approval from the [masked for review] Institutional Review Board, educators, family caregivers, and community members currently using FYBS were recruited using an electronic flyer that was disseminated on the FYBS email listserv, a pop-up on the FYBS website presented to site users, and posts on social media channels. To maximize participation, recruitment information was disseminated weekly throughout the recruitment and data collection periods. The recruitment flyer included a link to an online survey. Educators (e.g., teachers, school counselors, school psychologists), community providers, and family caregivers (N = 51) participated in the online survey between March and September 2023. Eligibility criteria included being 18 years of age or older and self-reported use of FYBS toolkit components. Of those, 29 respondents completed all required survey items (23 of whom also completed all open-ended responses), 11 provided partial responses, and 11 did not meet participation eligibility criteria (i.e., had not used components of the FYBS toolkit or were not 18 years of age or older). Only complete, eligible survey responses were included in analyses. The majority of survey participants identified as educators (n = 24), female (n = 24), and White (n = 19). Additional survey participant demographics can be found in Table 1.

Table 1

Table 1. Survey participant demographics.

The survey was also used to recruit current FYBS users for participation in a semi-structured interview. All interested survey respondents (n = 18) were invited to participate in an interview. Of those invited, five did not respond to communications from the research team to schedule an interview and one participant was excluded as they did not meet inclusion criteria, resulting in 12 interview participants. All participants identified as female, and the majority of participants identified as educators (n = 10). Interview participant demographics are available in Table 2.

Table 2

Table 2. Interview participant demographics.

2.2 Measures

2.2.1 Online survey

Eligible participants were first asked to report their primary role as a user of FYBS (educator, family caregiver, community provider, other), race, and ethnicity. Second, a modified version of the System Usability Scale (SUS; Brooke, 1996) was created to evaluate usability of FYBS. The SUS was developed in response to a need for a simple usability scale that aligns with industry standards outlined by the International Organization for Standardization (Brooke, 1996; Brooke, 2013; International Organization of Standardization, 2018). The SUS has demonstrated strong reliability and concurrent validity with other usability measures (Lewis, 2018). The SUS contains 10 items that are rated on five-point scale from “strongly disagree” to “strongly agree.” Ratings yield a composite score of overall usability that ranges from 0 to 100 (Lewis, 2018). To maintain the integrity of the SUS, only minor changes were made to the wording of each item and the directions. In addition, six optional open-ended survey questions were included about their use of FYBS: (1) What do you like best about FYBS? (2) What makes FYBS easy to use? (3) What makes FYBS difficult to use? (4) What suggestions do you have to improve FYBS? (5) What additional FYBS materials or supports would you like to have available? and (6) Do you plan to use FYBS again—how and why?

2.2.2 Semi-structured interview protocol

A semi-structured interview protocol was developed by the authors to align with the research questions. The semi-structured interview protocol included eight questions that focused on participants’ use of FYBS. These questions included preferences for different toolkit components, any adaptations made or desired, and additional resources that would be helpful to their implementation.

2.3 Procedures

Individuals who passed eligibility screening and consented to participate completed the online survey as described above. Participant responses to demographic items, modified SUS items, and open-ended items were de-identified prior to analysis. Participants were entered into a raffle to win one of eight $50 gift cards in appreciation of their time.

Interested survey participants were then invited to participate in a semi-structured interview. All interviews (n = 12) were conducted via a University-approved teleconferencing platform by one of two research assistants using the semi-structured interview protocol. Interviews lasted between 10 and 20 min, were audio recorded and then transcribed. Transcripts were de-identified and each participant was given a unique numerical code. In addition, participant quotations were carefully reviewed to ensure confidentiality. Participants were provided with a $50 gift card for their time.

2.4 Data analysis

An independent intramethod strategy (Moseholm and Fetters, 2017) was used to analyze data, meaning that we initially analyzed quantitative and qualitative data independently and then integrated findings to draw conclusions. Quantitative data collected from the online surveys were analyzed using descriptive statistics. Qualitative data (e.g., open-ended survey responses, interview data) were analyzed using thematic analysis and inductive coding. Braun and Clarke (2006, 2021) six-phase approach guided the thematic analysis: (1) becoming familiar with the data; (2) creating initial codes; (3) identifying themes or patterns; (4) reviewing potential themes; (5) defining and labeling themes; and (6) reporting findings. Inductive coding allowed codes to be developed from participants’ interview answers rather than any pre-specified codes generated by the research team. Two research assistants served as coders. They participated in a one-hour structured training that included a review of the research questions and procedures for conducting thematic analysis, including inductive coding (Braun and Clarke, 2006, 2021).

The two coders began by inductively coding the first participant’s transcript. Codes were generated about the participants’ perceptions of the usability of the FYBS toolkit, and if and how the participants adapted the toolkit to fit their needs. After coding the data related to the first participant, the first coder and second coder each wrote a memo detailing impressions and a description of the participant’s feelings about FYBS’s usability and what sorts of adaptations the participant made, if any, while using FYBS.

After coding the data for the first participant, the first and second coders met to discuss their interpretations and combine their codes. They repeated this process using the consolidated set of codes for the second and third participants’ data, creating additional codes as needed, and then for every two participants after that. After data for all 12 participants were coded, both coders reviewed the full dataset together to ensure all relevant data were coded.

Once initial coding was completed, the coders jointly organized the codes into clusters of related ideas to explore potential themes. Examples were included to help define and analyze what each potential theme included, and contradictory codes were placed within appropriate clusters and codes that did not fit into any potential themes on their own. Results of quantitative and qualitative analyses were then integrated to draw overarching conclusions.

2.5 Researcher reflexivity

All authors considered our positionalities related to the research questions and data throughout the course of the study. With respect to our gender and racial/ethnic backgrounds, the first, second, and third authors identify as cisgender (e.g., biological sex aligns with gender identity and expression) women, and the fourth author identifies as a cisgender man. The first, second, and fourth authors are White/non-Hispanic/Latinx, and the third author is White and Hispanic/Latina.

The second author is a co-creator of FYBS, and the first author is FYBS’ Implementation Coordinator. A graduate research assistant conducted all interviews to reduce the likelihood of social desirability bias (Nederhof, 1985; Bergen and Labonté, 2020). Although the first author supervised coding of the qualitative data, two research assistants new to FYBS were trained to code the data. The first three authors have worked in school settings as school psychologists or school psychology trainees, and the third author worked as a school support staff member. As such, the study and analysis were informed by our knowledge of child and adolescent mental health intervention and work in schools. Although this insight aided study design and data interpretation, we were careful to consider how participant’ experiences were unique and potentially different from our own. The first author and coders held regular meetings to discuss their interpretations of qualitative data and to review potential themes that emerged.

2.6 Measures to increase validity

We took several steps to increase study validity. The third and fourth authors documented coding decisions, consistencies and inconsistencies in the data, and the rationale for organization of the data in a series of memos. As presented in the results section, we included participant quotes throughout to allow participants to describe their experience with FYBS in their own words and to avoid deviating too far from the data. We also highlighted disconfirming evidence (i.e., evidence that contradicts or is inconsistent with other findings; Creswell and Miller, 2000) to ensure that results accurately depict participants’ perceptions of FYBS usability.

3 Results

3.1 Survey results

The Modified SUS provides a composite number representing the overall usability of FYBS. Across studies, average usability on the SUS has been calculated as a score of approximately 70 (Bangor et al., 2008; Sauro, 2011). A common practice among SUS users in research and industry is to interpret a mean SUS of at least 80 as indicating above average usability (Lewis, 2018). Participant responses (n = 29) yielded a mean SUS score of 82.5, which suggests participants rated FYBS usability as above average.

Descriptive statistics for each of the modified SUS items are available in Table 3 as the individual items can offer nuanced information regarding participant perceived usability in their context. In addition, the percentage of participants who indicated each level of agreement (Strongly Disagree to Strongly Agree) for the 10 modified SUS items can be found in Figure 1. Overall, participants overwhelmingly agreed that they felt confident using FYBS (93%), that most people would learn to use it very quickly (93%), that the functions in FYBS were well integrated (88%), that it was easy to use (90%), and that they would like to use it frequently (90%). The majority of participants did not feel that they needed to learn a lot before getting started with FYBS (86%), did not find it cumbersome to use (90%), did not find inconsistency in FYBS (86%), would not need the support of a technical person to use it (83%), and did not find it unnecessarily complex (83%).

Table 3

Table 3. Descriptive statistics for the modified system usability scale (SUS).

Figure 1

Figure 1. Percentage of participants who indicated each response option to all items on the Modified System Usability Scale (SUS).

Narrative comments provided in response to the open-ended survey items indicated that FYBS was easy to use, expressed that the children they used the toolkit with enjoyed it and that they enjoyed using FYBS, acknowledged minor adaptations made based on their individual contexts, and offered suggestions for the FYBS creators to consider in the development of future materials. A notable finding is that all respondents to the open-ended survey questions indicated that they would use FYBS again in the future. One participant responded that they would “definitely” use it again, elaborating:

The kids really respond to it. When I go into their classrooms with anything other than a FYBS lesson, they always ask when they will see the puppets. They also will say, “but we did not see the puppets today,” when I am leaving the room. Next year, I will use the videos we did not get to with the same group of kids (1st graders next year) to help encourage them to use the strategies they learned this year. I also plan to use the program with K next year as well.

3.2 Interview results

Across responses to semi-structured interview questions, usability and adaptation of FYBS can be summarized in five themes: (1) FYBS is easy to use, (2) children and (3) implementers both respond well to FYBS, (4) implementers personalize FYBS materials, and (5) considerations for implementation. A brief summary of supporting evidence for each theme is available in Table 4. Note that these themes were consistent with comments offered in response to the open-ended survey items, which is not surprising given that interview participants had also participated in the survey portion of the study.

Table 4

Table 4. Brief summary of emerging themes.

3.2.1 FYBS was easy to use

Users of FYBS found the program well-organized, the website easy to navigate, and the lessons simple and clear. The facilitators found that they could easily adapt FYBS into their learning environments and use the program consistently. One participant remarked:

I found the whole thing very easy to use, and I liked that it did not have to be taught in a linear progression. I knew that I only basically had six sessions, and so I just went through the strategies and picked out the six that I felt would be most beneficial for the group of kids I was working with. We covered one strategy per session, and it was just very easy to use.

Another shared: “The language, the ease of the website, the cue cards, etc. make this program very user friendly. It is stripped down to the basics and not over done with extra information or extraneous links.” Participants consistently noted the ease with which they learned about the program themselves and then were able to teach in their classrooms.

3.2.2 Children responded well to FYBS

According to participants, FYBS was effective for teaching emotion-focused coping strategies in ways that children responded well to and remembered. Facilitators shared that they felt the concepts were relevant and relatable to children, and that children were engaged in the lessons. Most participants expressed that the use of puppets seemed to increase effectiveness, that children enjoyed the strategies in the videos, and many described how children applied the strategies they learned about in other settings.

3.2.2.1 Puppets enhanced effectiveness

Many participants noted that the puppet characters and the use of the puppet-making activity enhanced the effectiveness of the lessons. For example, one participant shared: “…there’s just something about it with going a little bit more in depth, talking about each of the strategies, practicing the strategies with the puppets…I think it’s a little bit more powerful and practical than sort of just trying to introduce and teach coping skills.” One participant noted that children connected to the puppet characters featured in the videos: “The kids really connect to the puppets, and they know them by name, so they’ll say, ‘What’s Nico going to teach us today?’ ‘What are we going to learn from Mena today?’” Participants also discussed the contribution of the puppet characters and puppet-making component to improved engagement and communication. One shared: “I had read that a lot of times kids are more verbal and open, when sort of communicating through their puppets, and I definitely found that to be the case.” Overall, participants found that children connected to the characters in the videos and demonstrated increased verbal participation when they implemented the puppet-making activity.

3.2.2.2 Children enjoyed strategies

Participants found that children engaged in the lessons, enjoyed watching the videos and completing the lessons, and were particularly fond of specific strategies and videos. One participant noted: “Since we did only get through the first six, the first graders were saying, ‘Hey, can we…meet in second grade and do the rest of them?’ And so, we are planning to carry it over into next school year too.” Participants shared that children enjoyed the Calm Your Self strategies, which include Belly Breathing and Shake out the Yuck, the theme song, and the Catch Your Feelings strategy, Float Your Boat. Participants also noted that they observed the Connect With Others strategies to be helpful for children. In particular, many noted that Be a Kind Helper and Bring a High Five were easily understood by children, and that children enjoyed learning and using these strategies.

3.2.2.3 Children applied use of strategies across settings

Many participants found the FYBS strategies to be effective. Specifically, they noted increased coping strategies being used by children, that children remembered specific strategies, and that they would use the strategies, without prompting, throughout the day. For example, one participant shared the following anecdote:

So we did Float Your Boat two weeks ago, and then one of the boys, first grader, went up to the parent in the room and said, ‘I throw away my paper.’ And she was so confused, like, ‘What do you mean, you throw away your paper?’ He goes, ‘No, no, I did the Float My Boat. I wrote my angry feeling, and I crumpled it up and now I feel better! Yeah, unprompted and everything. He just did it, so that was so cool.

Overall, participants consistently found that children enjoyed the FYBS lessons, and that the lessons were memorable and useful.

3.2.3 Implementers responded well to FYBS

Across responses and data sources, participants indicated that FYBS was satisfying to use. Participants shared that they felt the lessons are thorough, they appreciated the supplementary tools for facilitating, and that FYBS was enjoyable for the implementer themselves. A survey participant indicated: “It was easy to explain to the children and the videos were enjoyable.” Another noted: “I love the videos and the handouts of how to make puppets.” Ultimately, participants shared that they enjoyed using FYBS with the children with whom they work.

3.2.4 Implementers adapted FYBS materials for personalized use and contexts

Many implementers of FYBS showcased flexibility in how FYBS could be used as they noted adaptations as to how the puppets were used, the way strategies were taught, the integration of supplemental activities, and to the materials. The specific adaptations of FYBS were strongly influenced by the needs of the children and the background of the implementer. As such, there was variation in the specific adaptations made by implementers.

3.2.4.1 Adaptations to puppet-making

Regarding adaptations made to the supplemental puppet-making activity, one implementer stated that children named their puppets and would talk to them. Other adaptations exhibited implementers and children using the puppets as an integral part of practicing FYBS strategies. One participant stated that the children used their own puppets to learn FYBS strategies, whereas another created “puppet centers” in the corner of their classroom in which children would talk with the puppets and practice FYBS strategies. In a similar vein, children in one classroom wrote and performed a play with their puppets in which the puppets would run into problems and would use a FYBS strategy. This participant shared how the puppets related to the children learning the strategies, “the kids could use their puppets, and the puppets could sort of teach the skill, and so we did incorporate that each time, and it felt like that was really good practice for the kids.”

3.2.4.2 Adaptations to teaching the FYBS strategies

Additional adaptations were made as to how FYBS strategies were taught. Some participants adjusted the strategies to apply to circumstances and emotions pertinent to the children receiving FYBS instruction. For example, strategies were used to manage interpersonal conflict and over-excitement. One participant reported integrating additional emotion identification content, including from other SEL programs, into their FYBS lessons. Implementers were able to adjust the way in which they taught FYBS such as through slideshow presentations, personalized discussion sheets, and individualized pacing of lessons. FYBS lessons were consistently shared with important adults in the children’s lives. As one participant shared, “I also ended up putting a sort of a booklet together of the tip sheets and videos and sending that home to parents. And I’ve already received feedback that parents are viewing them with their kids and working on the different strategies.”

3.2.4.3 Adaptations to supplement FYBS materials

FYBS materials were adapted by some implementers in a multitude of ways. These implementers printed out FYBS materials and presented it to children in a unique way. For instance, multiple implementers created notebooks with printed material including strategy cards and reflection sheets. In another classroom, FYBS tip sheets were printed and put into a “quiet corner” for children to have an area to use the strategies. In one instance, an instructor both used handouts provided by FYBS and created their own handouts that go along with FYBS.

In addition, many implementers integrated supplemental activities as an addition to FYBS. Most of the supplemental activities involved hands-on art projects related to concepts and strategies in FYBS. As one participant described, “I took the additional step to create art projects…to go along with the videos.” In some instances, picture books were read to go along with corresponding strategies. Additionally, other videos were used by an implementer to supplement FYBS content. This participant shared: “I love using picture books…as part of my guidance lessons, so I’ll find, um, and search for a picture book with the same theme to connect with it and then I use, um, like a little video I find on YouTube with a song that may connect to that theme, so I kind of make it a whole entire lesson.”

Overall, a diverse set of personalized adaptations to FYBS materials were noted. Both children and implementers were able to use FYBS in unique ways including how they integrated use of the puppets, the method of teaching the strategies, the supplemental materials used, and specific adaptations to the materials.

3.2.5 Considerations for implementation

Participants suggested a range of implementation considerations that could improve future usability. These suggestions fell within three sub-themes: (1) developmental appropriateness of materials and strategies; (2) puppet-making materials and guidance, and (3) possible additional and supplemental materials to support facilitator use.

3.2.5.1 Developmental appropriateness

Participants noted challenges with respect to the developmental appropriateness of certain FYBS strategies dependent upon a child’s age and prerequisite skills. To enhance developmental appropriateness for younger children and non-readers, one implementer suggested having increased visuals on the strategy cards that demonstrate how to perform a strategy. A concern among a few participants was that children had trouble differentiating between each strategy. One participant thought that some strategies and concepts were repetitive, and another brought up the need for prerequisite teaching of emotional identification skills. They elaborated, “I think one thing that I noticed that like, prerequisite skills before you could use this, they need to really know what their feelings are. So um, that is something that is either missing or it is something that you need to just make sure you have addressed, like, gone into like a lot more of that first.”

Two implementers indicated that Catch Your Feelings strategies were less relatable for children. One suggested that the Catch Your Feelings strategies were especially confusing for younger children. They shared: “they were thinking they were the strategies how to calm down versus how can I prevent myself from getting to that spot? So that was the only critique. I think it was a little confusing for five-year-olds.” As expanded upon by this implementer, the confusion seemed to be about the logic of Catch Your Feelings strategies preventing future negative emotions, especially for a younger age group.

3.2.5.2 Puppet-making materials and guidance

The lack of accessibility to puppet-making materials was the most frequently mentioned concern related to usability. Specifically, implementers most frequently cited large class sizes as a reason for not incorporating the puppet component of FYBS. On a similar note, some implementers expressed financial concerns as a reason why puppet activities were not carried out. As one implementer stated, “Because for me to make-, I have 75 kids in a grade, so that’s like 225 puppets to have to build. Um, and even making, um, paper bag puppets with the kids gets to be like wild and crazy and ends up being expensive too.”

Another concern that one implementer had was that they felt like it was unclear how to use the puppets. They articulated: “So sometimes some of the suggestions written in the facilitator tool book, I-, it just did not feel meaty enough, or like creative enough in some way, and also a little too abstract for me to understand how the kids were to use, maybe-, the puppets for certain techniques.”

A common suggestion to improve usability was to have puppet kits already made for their classrooms. As this implementer suggests, “I would love is if there was like, we could buy a puppet kit to like give the kids. So there was puppets that could be used instead of having to build them. Because that would be more financially accessible.”

A handful of individual participants shared unique desired additions to the puppet-making component. One shared that they wanted an option to purchase the three main puppet characters, another suggested a FYBS live performance, and a third expressed a desire for content that would show the puppet characters “growing up” along with the audience.

3.2.5.3 Potential additional materials to support facilitator use

Among the suggestions for additional strategies and topics, multiple participants wanted FYBS to generally have more videos and strategies. This sentiment is stated by one implementer, “Honestly, I’d love to see this as like a continuation curriculum. And like, I love using it as a supplemental resource and stuff like that, but I think having a larger curriculum, so like going throughout the year, and building upon these emotional regulation tactics, uh, I think would be really, really neat.”

Other video and strategy suggestions included content related to interacting with others such as interpersonal conflict, social situations, listening skills, and emotion identification in other people (e.g., for children who are neurodivergent). This implementer stated, “Now, what does your best self look like in social situations? What’s that like-, and maybe having the characters think about was I my best self sitting on the rug trying to listen to my teachers or was I, you know, annoying my friends? You know? Um, I think those would help kids understand the purpose of why they are in school.”

Most implementers also wanted more materials and resources. A number of participants indicated a desire for books that go along with themes in the FYBS videos. As one shared, “One thing some other programs do that I’ve seen there’s like-, with a video they’ll have like a little like printable book. So like-, like cartoon, like, pages and just going through like almost looking at like a social story or whatever happened. ‘So-and-so is angry because this happened. His friend reminded him to use a strategy.’”

Frequently, participants requested that additional supplemental material be added to FYBS. Specifically, many implementers suggested material that involved supplemental crafts. A few participants suggested the addition of corresponding activities to accompany each strategy. Some less frequent suggestions included posters that go along with each strategy, a sheet for each strategy, “play-based” material, explanations of psycho-educational content, guidance on Turn the Dial, and a video explaining the differences in strategy themes. Another suggested activity that was suggested was an in-person workshop with children. One implementer desired additional resources to support reinforcing children’s learning of the strategies. This participant elaborated: “Besides just watching the 2-min videos, more directions on things you can do to reinforce the lesson…I would just-, if there was a way to add in more description into that part, or information into that part, or just even more-, more like activities.”

4 Discussion

The present study explored users’ perceptions of FYBS usability along with adaptations made during implementation. Findings across both survey and interview results indicated that participants perceived FYBS to be usable. In particular, participants across methods reported FYBS as being effective, meaning that their intended goal of increased child emotion-coping was achieved. Participants also reported FYBS to be efficient in that they found it easy to use in their context, and that it required limited resources to use. Finally, participants indicated that FYBS was satisfying to use in that they enjoyed the components of the program and experienced positive affect themselves after using the program.

The incorporation of puppetry throughout FYBS was reported by implementers as enhancing the effectiveness of teaching the lessons, supporting findings by Kröger and Nupponen (2019) that puppets can promote student engagement and facilitate instruction. Participants reported observing increased use of emotion-coping strategies among children, with children using the strategies in other settings unprompted. These findings are consistent with previous literature that SEL programs increase SEL skills (Durlak et al., 2011; Cipriano et al., 2023). As previously stated, not only did participants report that the children enjoyed FYBS, but implementers also found FYBS satisfying to use. As found by Frenzel et al. (2009), teacher and student enjoyment in the classroom can be positively associated, meaning that there may be a relationship between student and implementer enjoyment of FYBS.

Participants personalized their implementation of FYBS, with adaptations made to puppet-making, teaching the strategies, supplemental activities, and to the materials. In other words, participants expressed that they were easily able to adapt all FYBS components to meet their needs. For example, participants integrated the puppets made as part of FYBS into classroom centers and other activities, integrated additional scenarios that were relevant for their context, created supplemental materials (e.g., worksheets, handouts), and facilitated additional activities (e.g., art projects) that related to FYBS strategies and concepts. The creativity and range of adaptations by the implementers were indicative of the intended flexibility of FYBS, which is consistent with findings by Koslouski et al. (2024).

Although these findings overwhelmingly indicate that FYBS was able to be used across contexts with effectiveness, efficiency, and satisfaction, participants did offer implementation considerations. One consideration focused on ensuring the developmental appropriateness of materials and strategies. For example, a few participants noted that some strategies may be less developmentally appropriate for young children, prerequisite skills may be needed before diving into strategy instruction, and increased visuals across materials could support understanding by non-readers. Another consideration related to the optional puppet-making component. Participants cited large class sizes and financial cost for puppet-making materials as considerations that may impact use of this FYBS component. In addition, many participants offered desired additions or supplements to the toolkit. These suggestions included additional materials and resources to support facilitator use that they hypothesized would enhance FYBS usability.

Overall, participant perceptions of usability were consistent in supporting high usability. Specifically, participants indicated that FYBS was easy to use, effective (e.g., children were engaged in teaching, children demonstrated independent use of coping strategies), and satisfying (e.g., implementers enjoyed teaching). Many similar themes related to adaptations made were noted across participants, yet each participant indicated targeting modifications based on child needs, personal teaching preferences, and their context.

4.1 Limitations

This mixed methods evaluation is not without limitations. Although irrelevant to the research questions, limited participant demographics were collected (i.e., role as implementer, gender identity, race, and ethnicity). In particular, collecting additional demographics around FYBS use (e.g., specific role as educator/community provider, age of children receiving FYBS instruction) may have allowed for additional interpretations regarding implementation context. The small number of participants is also a limitation, and as such, present findings may not generalize to all users of FYBS. In particular, the majority of these participants identified as educators (e.g., classroom teachers, school counselors, school psychologists), and therefore, more information is needed to better understand implementation and usability in non-education contexts (e.g., private practice, extracurricular/community-based programs, homes). Interview data that were collected, however, allowed for an initial exploration of general FYBS usability within different implementation contexts. Finally, it is possible that data were influenced by social desirability bias (Nederhof, 1985; Bergen and Labonté, 2020). The interview protocol asked specifically for downfalls (e.g., components that were less usable/interesting), but participants may have exaggerated the benefits or positive aspects of using FYBS.

4.2 Future directions and implications

Current findings offer meaningful next steps and implications for research and practice FYBS. With respect to future research, efficacy studies are needed to confirm implementer reports of positive child outcomes. In particular, given the variability in materials used and adaptations made, the specific components that contribute to effectiveness should be identified. For example, a number of participants indicated that the optional puppet-making component enhanced effectiveness. Others indicated that they did not use the reflection sheets yet still offered anecdotal reports of positive child outcomes. As with the findings presented in Koslouski et al. (2024), our findings add to reports of the joy experienced by implementers. This adds further justification to the need to explore intergenerational effects of FYBS, specifically with regard to child and adult co-regulation and emotional well-being. In addition, future research should address limitations of the present study, including increased identification of demographic variables of FYBS implementers, larger sample sizes to establish generalization of usability across contexts, and exploration of how FYBS is being used across types of implementers (e.g., educators, community providers, and families).

Findings also present considerations for practitioners interested in using FYBS. For one, findings indicate that FYBS is emerging as a usable toolkit for teaching emotion skills to elementary-aged children. The experiences shared by participants with respect to FYBS usability can be used to inform decisions about use in various settings. Adaptations made by participants to materials, procedures, and activities can also be used by future implementers in their contexts.

In terms of implications for FYBS expansion, participants offered suggestions that may be helpful for improving usability. For example, participants requested additional or supplemental materials and resources. Additional strategies focused on social and interpersonal skills and additional scenarios for older children were indicated as potential areas for expansion. With respect to materials, participants desired books that go along with the themes in the videos, pre-made puppet-making kits, posters and printed booklets of materials for their classrooms/offices, and other “play-based” materials. Participants also suggested adding resources such as workshops for both teachers and children, additional resources explaining the theory behind FYBS strategies, and additional guidance on instructional practices. Future directions for the creators of FYBS include incorporating these findings into revision of existing materials and creation of new materials.

Data availability statement

Some datasets presented in this article are not readily available given participant confidentiality in the qualitative responses. Requests to access the datasets should be directed to the corresponding author.

Ethics statement

The studies involving humans were approved by University of Connecticut Institutional Review Board. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

EI: Writing – original draft, Writing – review & editing. SC: Writing – original draft, Writing – review & editing. RT: Writing – original draft, Writing – review & editing. MW: Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. Feel Your Best Self was made possible through the generous support of Principal^® Foundation. Additional support has been provided by the Neag Foundation, which serves as a philanthropic force for positive change in education, health, and human services initiatives.

Conflict of interest

SC is a co-inventor of the Feel Your Best Self (FYBS) intellectual property owned by the University of Connecticut, is co-Chief Executive Officer with FYBS & Co, and she also has an equity interest (stock) in FYBS & Co.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^www.feelyourbestself.org

References

Agency for Healthcare Research and Quality (2022). 2022 National healthcare quality and disparities report: Child and adolescent mental health. Available at: https://www.ncbi.nlm.nih.gov/books/NBK587174/

Google Scholar

Agyapong, B., Obuobi-Donkor, G., Burback, L., and Wei, Y. (2022). Stress, burnout, anxiety and depression among teachers: a scoping review. Int. J. Environ. Res. Public Health 19:10706. doi: 10.3390/ijerph191710706

PubMed Abstract | Crossref Full Text | Google Scholar

American Academy of Pediatrics, American Academy of Child and Adolescent Psychiatry, Children’s Hospital Association . (2021). AAP-AACAP-CHA declaration of a national emergency in child and adolescent mental health. Available at: https://www.aap.org/en/advocacy/child-and-adolescent-healthy-mental-development/aap-aacap-cha-declaration-of-a-national-emergency-in-child-and-adolescent-mental-health/

Google Scholar

Bangor, A., Kortum, P. T., and Miller, J. T. (2008). An empirical evaluation of the system usability scale. Int. J. Human–Computer Interaction 24, 574–594. doi: 10.1080/10447310802205776

Crossref Full Text | Google Scholar

Bergen, N., and Labonté, R. (2020). "everything is perfect, and we have no problems": detecting and limiting social desirability bias in qualitative research. Qual. Health Res. 30, 783–792. doi: 10.1177/1049732319889354

PubMed Abstract | Crossref Full Text | Google Scholar

Brann, K. L., Daniels, B., Chafouleas, S. M., and DiOrio, C. A. (2022). Usability of social, emotional, and behavioral assessments in schools: a systematic review from 2009 to 2019. Sch. Psychol. Rev. 51, 6–24. doi: 10.1080/2372966X.2020.1836518

Crossref Full Text | Google Scholar

Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101. doi: 10.1191/1478088706qp063oa

Crossref Full Text | Google Scholar

Braun, V., and Clarke, V. (2021). Thematic analysis: A practical guide. London: SAGE.

Google Scholar

Brooke, J. (1996). “SUS: a “quick and dirty” usability scale” in Usability evaluation in industry. eds. P. Jordan, B. Thomas, and B. Weerdmeester (London, UK: Taylor & Francis), 189–194.

Google Scholar

Brooke, J. (2013). SUS: a retrospective. J. Usability Stud. 8, 29–40.

Google Scholar

Brush, K. E., Jones, S. M., Bailey, R., Nelson, B., Raisch, N., and Meland, E. (2022). Social and emotional learning: from conceptualization to practical application in a global context. Life Skills Educ. Youth: Critical Perspect. 5, 43–71. doi: 10.1007/978-3-030-85214-6_3

Crossref Full Text | Google Scholar

Cipriano, C., Strambler, M., Naples, L., Ha, C., Kirk, M., Wood, M., et al. (2023). The state of evidence for social and emotional learning: a contemporary meta-analysis of universal school-based SEL interventions. Child Dev. 94, 1181–1204. doi: 10.1111/cdev.13968

Crossref Full Text | Google Scholar

Collaborative for Academic, Social, and Emotional Learning . (2013). SEL in the classroom. Available at: https://casel.org/systemic-implementation/sel-in-the-classroom/ (Accessed December, 2023)

Google Scholar

Creswell, J. W., and Miller, D. L. (2000). Getting good qualitative data to improve educational practice. Theory Pract. 39, 124–130. doi: 10.1207/s15430421tip3903_2

Crossref Full Text | Google Scholar

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., and Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev. 82, 405–432. doi: 10.1111/j.1467-8624.2010.01564.x

PubMed Abstract | Crossref Full Text | Google Scholar

Fetters, M. D., Curry, L. A., and Creswell, J. W. (2013). Achieving integration in mixed methods designs—principles and practices. Health Serv. Res. 48, 2134–2156. doi: 10.1111/1475-6773.12117

PubMed Abstract | Crossref Full Text | Google Scholar

Frenzel, A. C., Goetz, T., Lüdtke, O., Pekrun, R., and Sutton, R. E. (2009). Emotional transmission in the classroom: exploring the relationship between teacher and student enjoyment. J. Educ. Psychol. 101, 705–716. doi: 10.1037/a0014695

Crossref Full Text | Google Scholar

Gross, J. J. (2014). Handbook of emotion regulation. 2nd Edn New York: Guilford Press.

Google Scholar

Hossain, M. M., Nesa, F., Das, J., Aggad, R., Tasnim, S., Bairwa, M., et al. (2022). Global burden of mental health problems among children and adolescents during COVID-19 pandemic: An umbrella review. Psychiatry Res. 317:114814. doi: 10.1016/j.psychres.2022.114814

Crossref Full Text | Google Scholar

International Organization of Standardization . (2018). ISO 9241-11:2018(en) Ergonomics of human-system interaction: Part 11: Usability: Definitions and concepts. Available at: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en [Accessed January, 2024]

Google Scholar

Iovino, E. A., Koslouski, J. B., and Chafouleas, S. M. (2021). Teaching simple strategies to foster emotional well-being. Front. Psychol. 12:772260. doi: 10.3389/fpsyg.2021.772260

PubMed Abstract | Crossref Full Text | Google Scholar

Jones, S. M., Brush, K. E., Ramirez, T., Mao, Z. X., Marenus, M., Wettje, S., et al. (2021). Navigating SEL from the inside out: Looking inside and across 33 leading SEL programs – A practical resources for schools and OST providers. The EASEL Lab, Harvard University. Available at: https://www.wallacefoundation.org/knowledge-center/Documents/navigating-social-and-emotional-learning-from-the-inside-out-2ed.pdf

Google Scholar

Kobylińska, D., and Kusev, P. (2019). Flexible emotion regulation: how situational demands and individual differences influence the effectiveness of regulatory strategies. Front. Psychol. 10:72. doi: 10.3389/fpsyg.2019.00072

PubMed Abstract | Crossref Full Text | Google Scholar

Koslouski, J. B., Iovino, E. A., and Chafouleas, S. M. (2024). Feel your best self: insights from elementary teachers' use in teaching emotion-focused coping strategies. Social Emotional Learn.: Res. Prac. Policy 3:100037. doi: 10.1016/j.sel.2024.100037

Crossref Full Text | Google Scholar

Kröger, T., and Nupponen, A. M. (2019). Puppet as a pedagogical tool: a literature review. Int. Electronic J. Elementary Educ. 11, 393–401. doi: 10.26822/iejee.2019450797

Crossref Full Text | Google Scholar

Lewis, J. R. (2018). The system usability scale: past, present, and future. Int. J. Human–Computer Interaction 34, 577–590. doi: 10.1080/10447318.2018.1455307

Crossref Full Text | Google Scholar

Moseholm, E., and Fetters, M. D. (2017). Conceptual models to guide integration during analysis in convergent mixed methods studies. Methodolog. Innovat. 10:205979911770311. doi: 10.1177/2059799117703118

Crossref Full Text | Google Scholar

Nederhof, A. J. (1985). Methods of coping with social desirability bias: a review. Eur. J. Soc. Psychol. 15, 263–280. doi: 10.1002/ejsp.2420150303

Crossref Full Text | Google Scholar

Sanetti, L. M. H., and Collier-Meek, M. A. (2019). Supporting successful interventions in schools: Tools to plan, evaluate, and sustain effective implementation. New York: Guilford Press.

Google Scholar

Sauro, J. (2011). A practical guide to the system usability scale: Background, benchmarks & best practices. Denver: Measuring Usability LLC.

Google Scholar

Thomas, R. (2024). Almost half a million children waiting to start mental health treatment, new figures show. The Independent. Available at: https://www.independent.co.uk/news/health/children-mental-health-autism-nhs-b2482324.html.

Google Scholar

UNICEF . (2019). Increase in child and adolescent mental disorders spurs new push for action by UNICEF and WHO. Available at: https://www.unicef.org/press-releases/increase-child-and-adolescent-mental-disorders-spurs-new-push-action-unicef-and-who.

Google Scholar