This article is part of the Research TopicAdvancing inclusive education for students with special educational needs: Rethinking policy and practiceView all 7 articles
Developing a participatory research framework through serious games to promote learning for children with autism
1Computer Science Division, Department of Mathematics, Faculty of Science, Ain Shams University, Cairo, Egypt
2Department of Psychological Studies, Faculty of Postgraduate Studies for Childhood, Ain Shams University, Cairo, Egypt
People with autism, or Autism Spectrum Condition (ASC), is becoming increasingly common worldwide. Since individuals with ASC vary in their skills and methods that work for one may not work for another, many technology designers find it challenging to engage effectively with this population. Serious games (SGs) offer an intelligent learning environment that supports lifelong learning for individuals with ASC. Despite the availability of several frameworks, the question of whether SGs for individuals with ASC can have a dedicated framework remains unresolved. The objective of this study is to create a general framework for the design of serious games that can be applied to a variety of SGs targeting individuals with autism. A new participatory research framework is presented to assist game designers and relevant stakeholders in developing effective SGs for people with ASC. Through participatory sessions and a design thinking process, this framework seeks to involve users and relevant stakeholders as “design partners” in the design process. The framework was employed in the development of a new SG, called SALY (Simulation, Attention, Learn, and PLAY), designed to improve attention span and emotion recognition in individuals with ASC. Three research questions are discussed, and the mixed-methods approach adopted for the investigation. Several usability metrics were used to evaluate the game’s effectiveness, efficiency, and user satisfaction. The results show that the proposed game holds significant potential and will be of interest to educators and learners alike.
1 Introduction
Participatory research (PR) has gained prominence in pedagogical research, particularly in the field of special educational needs. PR involves systematic inquiry conducted in close collaboration with individuals affected by the research topic, with the goal of initiating action or change (Vaughn and Jacquez, 2020). Unlike traditional research, PR prioritizes co-constructing research through partnerships between end users, communities, and other stakeholders affected by the issue being studied (Vaughn et al., 2018). Key et al. (2019), for example, describe research engagement as ranging from community-informed to community-driven. Recently, several studies have encouraged researchers to move beyond merely gathering children’s opinions to more meaningful engagement (e.g., Bakhtiar et al., 2023; Scott-Barrett et al., 2023; Kelly et al., 2023; Kay et al., 2023; Simpson et al., 2022; Anselma et al., 2020; McVeety and Farren, 2020). Bakhtiar et al. (2023) found that 23 out of 25 studies reported a wide range of benefits from conducting research with children, including authentic and meaningful participation, greater understanding of end users’ rights, and personal characteristics such as increased confidence, well-being, and sense of agency. Social benefits were also noted, such as collaboration, leadership, inclusivity, and contributing to the community. The knowledge children possess was valued and used as a foundation for research and planning. According to Vaughn and Jacquez (2020), researchers across disciplines view PR as a collaborative inquiry process that extends beyond knowledge generation to achieving real-world impact. They provided a comprehensive list of participatory research frameworks, orientations, and approaches. Building on these frameworks, our study rethinks the role of users in research, shifting them from informants to active participants in the design process. By involving users’ voices throughout the entire research project, not just at the results stage, we present an innovative participatory research framework. This framework aims to support experts, educators, and designers in creating effective SGs for individuals with autism and provides practical guidance to practitioners, educators, and other stakeholders on the benefits of involving users and related parties in the planning and design phases of research. Autism, also known as Autism Spectrum Condition (ASC), is a neurodevelopmental disorder that can negatively affect children’s development (Bottema-Beutel et al., 2021). It is a lifelong developmental disability, often accompanied by learning difficulties, that impacts how people learn, behave, communicate, and interact with others. Individuals with ASC may experience difficulties in various areas of daily living, including emotion recognition (Glumbic et al., 2022). Since autism is a spectrum condition, its effects on individuals vary widely. The term “spectrum” refers to the broad range of behaviors and challenges faced by individuals with ASC. According to the Diagnostic and Statistical Manual of Mental Disorders (DSM-5, 2013), individuals with autism often exhibit deficits in social interaction, social communication, and imagination. These deficits lead to difficulties in attention to social inputs and in understanding the emotions and actions of others, which can hinder learning. Hobson (1993a, 1993b) highlights the challenges people with ASC face in understanding complex emotional expressions, pointing to a fundamental deficiency in interpersonal connections.
2 Problem statement, objectives and research questions
Human emotion recognition plays a central role in everyday life. An emotion recognition task is a powerful and useful technique for assessing human emotional states. Generally, this task is easy for neurotypical individuals, but it becomes more challenging when considering children with ASC. Since children with ASC tend to prefer interacting socially with objects rather than people, a growing number of studies have examined the effectiveness of digital technology (DT) for people with ASC, including serious games (SGs) (e.g., Gallud et al., 2023; de Carvalho et al., 2023; Abd El-Sattar, 2023b; Pavez et al., 2023; Tsikinas and Xinogalos, 2020), virtual agents (e.g., Abd El-Sattar, 2023a), immersive technologies (e.g., Tene et al., 2024; Tang et al., 2022; Abd El-Sattar, 2024), robots, and more. In the context of disability, the use of social robots, particularly with children diagnosed with autism spectrum conditions (ASC), may help foster the development of novel social behaviors and enhance skills in areas of difficulty (Ferrari et al., 2009; Conti et al., 2015). According to Dunst et al. (2013), robots can act as social mediators, helping children with autism engage with humans after they initially interact with the machine. Baron-Cohen's (2009) empathizing-systemizing (ES) hypothesis suggests that people with autism often prefer interacting with formal, predictable systems that have clear engagement rules. With sufficient programming, robots can be tailored to meet the specific needs of children with ASC, creating predictable social scenarios that reduce anxiety and fear. On the other hand, serious games (SGs) create a smart learning environment that supports individuals with ASC in pursuing lifelong learning, offering a new paradigm for education. Interacting with robots and/or SGs has various benefits for children with ASC, including increased social acceptability (Dunst et al., 2013), imitation-based motor communication (Duquette et al., 2008), and the ability to maintain shared attention (Robins et al., 2004). SGs are educational multimedia tools designed to help learners acquire specific skills. Through gameplay, learners can develop certain competencies or knowledge as part of the educational strategy. This approach transforms academic material into a game format, making learning more engaging and interactive (Fedwa et al., 2014). However, engaging individuals with autism in the development of digital technologies, such as SGs, poses challenges due to the diversity and uniqueness of each person’s needs and talents, which may change over time. This variability makes it difficult for technology designers to create effective, personalized solutions. As a result, autistic children’s perspectives are often overlooked in research. Furthermore, most studies in this field focus on guidelines and decisions rather than on creating a design framework for SG development. Despite the existence of several frameworks, the question of whether SGs for individuals with ASC should have their own framework remains unresolved. This research aims to address these challenges by proposing a general framework for SG design that can be applied to a wide variety of SGs targeting individuals with autism. It emphasizes the importance of providing autistic children with practical and meaningful opportunities for interaction and promoting their voices. An innovative participatory research framework is presented for developing effective SGs for individuals with ASC, with the goal of improving their attention span and emotion recognition abilities. The framework combines design thinking with participatory design sessions to generate creative solutions that meet users’ needs. In a case-based learning research approach, the framework was applied to the development of a new SG called SALY (Simulation, Attention, Learn, and PLAY). The SALY game is designed to help individuals with autism enhance their attention span and emotion recognition abilities by blending technology with learning. It also served as a tool to investigate the research questions presented in Table 1.
Table 1
Table 1. The study’s research questions (RQs).
3 Literature review
This section outlines some of the prior research on the use of serious games (SGs) for teaching and enhancing emotion recognition in individuals with different learning disabilities, including Autism Spectrum Condition (ASC):
1. LIFEisGAME (Alves et al., 2013) is an iPad prototype with five game modes: Build the Face, Recon Mee Match, Recon Mee Free, Sketch Mee, and Memory Game, designed to improve autistic children’s ability to recognize faces and emotions. The analysis indicates that while LIFEisGAME is visually appealing and entertaining, therapists suggested several modifications for each game mode. These include adding more customization options, increasing levels of difficulty, incorporating musical stimuli to boost motivation and feedback perception, and providing clearer instructions on how to play the game.
2. JeStiMulE (Serret et al., 2014) is a serious game designed to teach emotion recognition to children with both low and high-functioning autism. It consists of three phases: calibration, learning, and training. The learning phase includes three levels of increasing complexity: recognizing faces, faces with gestures, and faces with gestures and verbal cues. One limitation highlighted by the authors is the lack of a control group, as the study was exploratory and primarily aimed at identifying potential JeStiMulE users. However, descriptive analyses showed that JeStiMulE is adaptable, effective, and efficient in teaching emotion recognition to individuals with autism. A more recent review of JeStiMulE by Elhaddadi (2022) and Elhaddadi et al. (2021) confirmed the effectiveness of the game for addressing deficits in emotional facial expression (EFE) recognition among children with autism.
3. Emotiplay (Fridenson-Hayo et al., 2017) is an engaging and successful psycho-educational intervention that was used to teach emotion recognition (ER) to children with high-functioning autism across three distinct cultures. This was achieved through the interpretation of facial expressions, speech patterns, and body language, and the integration of these insights into context using Emotiplay’s serious game (SG). As a computer-based intervention, Emotiplay’s SG demonstrates cross-cultural benefits. The study highlighted two key design elements that enhance user motivation and improve learning outcomes: the use of a storyline with goal-directed behaviors, and increased gaming elements that facilitate the transfer of learning.
4. EmoStory (Min et al., 2018) is a game-based interactive narrative system designed to support children’s emotional development. It includes three narratives set in different contexts (school, home, and park), which use animations and emotional sounds to help autistic children understand six emotions, associated contexts, and facial expressions. The game features multi-level challenges embedded within the narrative, allowing autistic children to practice step-by-step, and provides real-time feedback based on facial expression recognition and visual cues to assist in making facial expressions.
5. JEMImE (Grossard et al., 2019) is a serious game aimed at helping autistic children learn to produce emotions such as happiness, anger, and sadness in a 3D virtual environment with social contexts. The game has two phases: training and playing. In the training phase, two imitation games and two emotional production games are used to teach participants to make adaptive facial expressions (FE). Each game has two variants—one with a less confusing design and without emotionally charged background images, and the other with a social context. The authors concluded that JEMImE has significant potential for supporting autistic children’s emotional development.
6. ALTRIRAS (Almeida et al., 2019) is a role-playing game (RPG) designed to assist children with autism in recognizing facial expressions associated with four basic emotions: joy, sadness, anger, and surprise. Unlike competitive games, ALTRIRAS focuses on social interaction. The game offers a variety of puzzle-solving tracks and follows the adventures of two characters, Rex and Tina, as they teach another character, Emotion. Players take on a central role in the story and gain abilities as they progress through the plot. The game aims to improve autistic children’s communication, perception, and quality of life.
7. GDF (Tsikinas and Xinogalos, 2020) proposes a serious games design framework (GDF) to help experts, special education instructors, and designers create effective SGs for individuals with ASC and intellectual disabilities (ID). The GDF consists of three main components: learning content and game mechanics, assessment, and pedagogy. The pedagogical component includes learners, educators, participatory design, and learning objectives. Additional game elements, such as self-learning, immersion, and continual challenge, are integrated to create an immersive experience that supports learners with ASC and ID in achieving their learning goals.
4 Contribution, discussion of research questions and methodology
Our study has made the following contributions to address the challenges of technology development for individuals with ASC, in response to the research questions outlined in Table 1:
1. New Participatory Research Framework: We developed a participatory research framework to assist game designers and relevant stakeholders in creating effective serious games (SGs) for individuals with ASC. Through participatory sessions and a design thinking process, the framework actively involves users and relevant stakeholders as “design partners” in the design activities.
2. Development of SALY: Using case-based learning research, the framework was applied to develop a new SG called SALY. This game helps individuals with ASC recognize facial expressions related to six basic emotions while also improving their attention and observation skills. SALY incorporates four mini-game engines: Simulation, Attention, TASALY, and Matching, which are based on both visual and auditory stimuli. The fantasy music-based social narrative game, TASALY, provides intrinsic rewards, while extrinsic rewards like points, levels, and badges enhance motivation and flow, keeping learners emotionally engaged, satisfied, and immersed in the game.
3. Advanced Techniques in Simulation Engine: To support individuals with ASC in learning how to express their emotions, we employed advanced computer graphics and machine learning techniques in the development of the simulation mini-game engine.
4. Usability Evaluation: We used a variety of usability metrics to evaluate the proposed game’s effectiveness, efficiency, and user acceptance. Comparisons were made with existing techniques, and the game was tested with a wide range of users, including those without impairments. A mixed-methods research approach was used to gather and analyze the data.
5. Technology Acceptance Model (TAM): To assess the level of acceptability of the technology, we applied the Technology Acceptance Model (TAM) proposed by Davis (1989).
6. The research questions raised in Table 1 are reviewed below, followed by a discussion of the research methodologies and methods employed to address them.
4.1 Discussion of RQ1 and the proposed framework
Developing serious games (SGs) is a complex task, requiring the integration of four primary components: learning elements (pedagogy and educational content), game design (fun), technology, and learning theories (Abd El-Sattar, 2023b). Each component demands collaboration among various stakeholders and specialists with diverse expertise, a process often facilitated through the participatory design (PD) method. The paradigm for design research has recently shifted from a user-centered approach—where the user is treated as the subject—to a participatory one, where the user is a partner (Maun et al., 2023; Wohofsky et al., 2023; Kinnula and Iivari, 2021; Luck, 2018). Bakhtiar et al. (2023) identified three primary types of engagement strategies in 25 studies: (a) child-led research, (b) children as co-researchers, and (c) youth participatory action research. Our approach, inspired by these findings, significantly rethinks the role of users—transforming them from mere informants into active partners in the design process. Table 2 outlines a set of design guidelines along with detailed explanations, which form the foundation of our proposed framework. The framework consists of three core parts: pedagogy, educational content and game characteristics, and assessment, as illustrated in Figure 1. The initial step in the proposed framework focuses on the pedagogical elements of SGs, specifically participatory design and design thinking. Once these pedagogical elements are determined and agreed upon, game characteristics and educational content are applied and organized into three key attributes: game elements, game aesthetics, and user experience. After the game is developed, an evaluation process is conducted to determine whether the prototype meets the desired objectives. Feedback from this phase is used to update instructions and review any elements that did not result in the intended learning outcomes. This process continues iteratively until all goals and objectives are met.
Table 2
Table 2. A lists of design principles for individuals with ASC, along with an explanation.
Figure 1
Figure 1. The proposed framework.
4.1.1 Participatory design (PD) and participatory sessions
The diversity among individuals with Autism Spectrum Condition (ASC)—in terms of their unique needs and evolving talents—makes it challenging for technology designers to engage with them effectively. Technology designers often struggle to address the entire system when developing solutions for ASC, making a community-driven design approach necessary. Participatory Design (PD) offers a solution by involving users and relevant stakeholders in the design process. PD is a design methodology that integrates users in every stage, from ideation to prototype testing (Maun et al., 2023; Wohofsky et al., 2023; Luck, 2018). By engaging as “design partners,” users can provide valuable insights into the tasks, context, and expected behavior of the future system. Stakeholders also play an essential role, representing the various adults and people in the user’s life. We categorized stakeholders based on the system proposed by Borjesson et al. (2015) as follows:
1. User/Participant: Individuals or children who interact with the system or technology developed.
2. Proxy: The user’s immediate environment, including parents, teachers, and supporters, who speak on their behalf.
3. Expert: Specialists such as therapists, psychologists, or teachers who assist in the design process. Unlike proxies, they provide insights representing groups of children rather than individual experiences.
4. Facilitator: Adults who help establish relationships between children and researchers/designers and provide practical support during activities.
The participatory sessions, a form of group work, proved invaluable in assisting the design process. These sessions validated initial design proposals, gathered new ideas, facilitated the understanding of user needs, and assessed which features foster higher motivation and engagement. Users were able to act as informants throughout the design process. Our participatory sessions followed a multi-stage process inspired by the Bluebells process (Kelly et al., 2006). The workflow was iteratively developed based on feedback from relevant stakeholders, including users. Figure 2 illustrates the steps of the participatory sessions, based on the Bluebells process:
Figure 2
Figure 2. Workflow for PD Sessions.
[1] Stage #1 (Before Play):
The design team conducts fact-finding and identifies required activities, which are validated by experts. The goals at this stage are:
A. A better understanding of system objectives, tasks, user needs, preferences, and context of use. Meetings were held with experts (e.g., psychologists, psychiatrists) to identify goals and gather user data through qualitative and quantitative methods (e.g., surveys, interviews, focus groups).
B. Visualizing the final system through design elements such as characters, avatars, scenarios, and storyboards. Scenarios illustrate intended user behavior and specific use cases.
[2] Stage #2 (During Play):
Participants engage in design activities. At the start of this phase, users are introduced to the system’s objectives and activities by a therapist. Participants are then assigned tasks that align with the session’s goals.
[3] Stage #3 (After Play):
After the sessions, the design team compiles and analyzes the results. The final evaluation process takes place, with each participant’s game data being gathered for analysis to confirm usability.
4.1.2 Design thinking (DT)
The Design Thinking (DT) technique was integrated with participatory sessions to dismantle traditional barriers and produce research more aligned with the objectives and preferences of the autistic community. DT is a methodology that connects user needs with what is technically feasible and practical, utilizing strategies such as participation, co-design, co-creation, and intuitive problem-solving (Braun and Clarke, 2022; Fabri et al., 2016; Koh et al., 2015). At its core, DT emphasizes empathy—the capacity to understand and solve problems from the perspective of another individual, according to Efilti and Gelmez (2023). Empathy enables designers to step into another person’s shoes, comprehend their situation, and devise solutions that address their challenges. Several frameworks exist to implement the DT process, including:
• Empathize-Define-Ideate-Prototype-Test (EDIPT) process (5-step).
For this study, we employed the 5-step EDIPT process, which was thoroughly explored by Fabri et al. (2016) and is illustrated in Figure 3.
Figure 3
Figure 3. The 5-step empathize-define-ideate-prototype-test (EDIPT) DT process.
4.1.3 Research methods
A research method is a procedure for generating or gathering data. It is important to note that mixed-methods research was employed to collect and evaluate the data for this study. We utilized a mixed-methods approach (qualitative and quantitative) to examine the topic from multiple perspectives, offset the limitations of one data type with the strengths of the other, and enhance the evaluation by combining the advantages of both methods. Our findings build upon those reported by Pettersson et al. (2018) and Borjesson et al. (2015).
4.2 Discussion of RQ2
The use of ‘technology-enhanced learning,’ along with game elements, attributes, user experience, accessible design, and assessment, presents promising research avenues for educating individuals with ASC. The concept of ‘technology-enhanced learning’ emphasizes the importance of leveraging technology to improve learning outcomes. In our study, we utilize machine learning and computer graphics techniques to assist individuals with ASC in expressing emotions and accurately recognizing facial expressions associated with the six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. This is achieved through the implementation of a simulation mini-game engine, which is discussed in greater detail in the ‘System Development and Implementation’ section.
4.3 Discussion of RQ3
Initially created by Davis (1989), the technology acceptance model (TAM) was used to assess the degree of acceptability of the technology in use. In terms of usability testing, a number of usability metrics were used to assess and evaluate the functionality and performance of the proposed application, including the Fun Toolkit (Gavin and Matthew, 2012) and the System Usability Scale (SUS) (McLellan et al., 2011). A thorough description of each of these elements may be found in the section under “Assessment, Usability Testing, and Obtained Outcomes.”
5 System development and implementation
5.1 System architecture and design
A graphical overview of the SALY system architecture is presented in Figure 4, while a complete navigation scheme of the developed system is shown in Figure 5. The first level displays the primary user interface, and the second level showcases the various modules of the system. This level includes the registration page, login screen, play mode, assessment file folder (which stores game data for each player throughout the gaming experience), and the logout screen. Registration occurs only during the first time participants access the system, during which they fill out a form requesting their name, age, and other relevant information. After registering, participants log in to the system using the username and password they previously created. The assessment file folder is utilized to monitor each participant’s results and track their learning progress. The third level provides a general overview of the four mini-game engines, each featuring a different level of difficulty included in the play mode.
Figure 4
Figure 4. SALY’s system architecture.
Figure 5
Figure 5. SALY’s system navigation scheme architecture.
5.2 System modeling
System modeling is the process of creating abstract representations of a system, with each model offering a unique perspective. The Unified Modeling Language (UML) (Rafael and Nuria, 2016) is a type of graphical notation used for system modeling. UML includes several diagram types, which are divided into two categories: structure diagrams and behavior diagrams. Structure diagrams, such as class and package diagrams, represent the static structure of the system, whereas behavior diagrams depict the dynamic behavior of objects within the system, illustrating a sequence of changes over time. Common types of behavior diagrams include use case, state, activity, and sequence diagrams. A use case diagram describes a system’s functional requirements in terms of use cases. It simulates the system’s intended functionality (use cases) and its environment (actors), illustrating how the system interacts with its surroundings. Appendix A provides illustrated examples of the four different types of behavior diagrams—use case, state, activity, and sequence—for the SALY system.
5.3 Simulation game engine implementation
This subsection provides a comprehensive explanation of the answer to RQ2 (Table 1). A recent branch of modern machine learning (ML) research called deep learning (DL) learns features and tasks directly from data (e.g., images, text, or sound). One of the most frequently applied DL networks for various classification problems is the convolutional neural network (CNN) (e.g., Alzubaidi et al., 2021; Samar et al., 2023a; Samar et al., 2023b). The success of CNNs is largely due to their inherent ability to automatically extract features from input data without requiring operator intervention (Li et al., 2021). We employed CNNs for emotion detection in the simulation game engine implementation. The implementation process comprises a feature extraction subsystem and a neural network that acts as a classifier subsystem, as illustrated in Figure 6. The role of the CNN as a classifier is to take the input image and output the probability of it belonging to a specific class (anger, disgust, fear, happiness, sadness, or surprise). Our simulation game engine is based on a CNN that was trained using the Radboud Faces Database (Langner et al., 2010), which is publicly available. The CNN analyzes an image as input and employs a strategy to predict the corresponding emotion as output. As demonstrated in Figure 7, the CNN architecture generally consists of four layers: input, convolution, pooling (sub-sampling), and fully connected. The input layer stores the pixel values of the input image. The image is divided into receptive fields that feed into the convolutional layer. Receptive fields are the areas of the visual field where a single neuron is activated in response to a stimulus. Features from the input image are extracted using the convolution layer. Convolution layers are based on the mathematical operation called “convolution,” performed on two variables (f*g) to produce a third variable. Figure 8 visually depicts a convolutional layer. Before training the network, a group of parameters called hyper-parameters is associated with the convolution layer, including learning rate, filter size, stride, activation function, and zero-padding (Alzubaidi et al., 2021). The stride refers to the number of units by which the filter slides over the input image. Hyper-parameters are constants whose values must be determined before building the models. The stochastic gradient descent (SGD) optimization algorithm (Habib and Qureshi, 2022) and its variants are the most widely used algorithms for training CNNs. The goal of any optimization problem during neural network training is to find the ideal weights and biases that minimize the cost—also referred to as loss or error—where a larger cost indicates a less efficient network. The amount by which the weights are updated during training is referred to as the “learning rate (LR).” The learning rate is a critical hyper-parameter that cannot be defined through explicit formulation and should be carefully considered during the training process. Typically, this parameter is determined through trial and error, with some researchers setting it to a constant value [e.g., 0.01, as in Pavez et al., 2023]. Joshi et al. (2019) indicate that even a small variation in the learning rate can significantly impact the network’s convergence, learning speed, and overall performance. Therefore, to enhance and accelerate the learning process, we employed a cyclic learning rate throughout the training phase, as described in Samar et al. (2023b). Convolution produces “feature maps,” which are collections of various features. Consequently, a pooling layer is employed to reduce the dimensionality of each feature map while retaining the most critical data. Max pooling, min pooling, and average pooling are three commonly used pooling techniques. A visual representation of the pooling layer using the max pooling technique is depicted in Figure 9. Before the convolutional layer, the system had been performing computations in a linear fashion. The selection of an appropriate activation function—such as Sigmoid, Tanh, Rectified Linear Unit (ReLU), or variants of ReLU (e.g., Leaky ReLU and PReLU)—introduces non-linear combinations of features (Alzubaidi et al., 2021). In our implementation, we used ReLU to expedite and enhance training by retaining positive values and mapping negative values to zero. The final layer, used for classification tasks, is fully connected and is responsible for performing a global operation by taking input from all the various feature extraction stages and conducting a global analysis of the outputs from all the preceding layers. Given that there are six basic emotions, its purpose is to receive an input volume and produce an N-dimensional vector, where N equals six (anger, disgust, fear, happiness, sadness, and surprise).
Figure 6
Figure 6. The simulation game engine workflow.
Figure 7
Figure 7. The CNN general architecture for emotion detection.
Figure 8
Figure 8. A visual representation of a convolutional layer.
Figure 9
Figure 9. A visual representation of CNN using the max pooling technique [as presented by Li et al., 2021].
6 The SALY game’s design elements
6.1 Design elements
The main elements of the SALY game design are:
• Acts: It is defined as the highest-level element in the SALY game, which structures it into different parts. This structure includes the main user interface, registration page, play mode, and assessment mechanism for monitoring participants’ progress, among other features. The choice of language, gameplay, and goal setting are determined based on the specifications and directions provided by the therapist or parent.
• Scenes: The gameplay is structured into acts, with each act further divided into scenes. Scenes occur within one or more scenarios, each presenting varying levels of difficulty. Each scene comprises several actions that outline the events taking place in the SALY game. For example, the following actions are included within the simulation game:
o Set the Emotion Character → Associated Action.
o Simulate a Character’s Emotion Visually → Challenge Action.
o Count the Success and Failure → Count Action.
o Timer Control → No Associated Action.
o Rewards → Two types of rewards: (1) Extrinsic like points and badges, and (2) Intrinsic for creating emotions among participants by playing TASALY & Matching games.
• Characters: To facilitate stimulation and emotion recognition, SALY features a variety of characters, including photographs of human faces, human cartoon faces, and emojis (Bai et al., 2019). Emojis, derived from the Japanese words “e” for “picture,” “mo” for “write,” and “ji” for “character,” are utilized to convey and enhance feelings. The game’s startup user interface includes a tutorial featuring an animated human cartoon face displaying various emotions, which combines visual and auditory stimulation to engage users and help them understand the necessary tasks (see Figure 10).
Figure 10
Figure 10. A snapshot of an animated human cartoon face with different emotions.
6.2 SALY gameplay
Gameplay defines how the learner and the game interact with each other. It simply means playing the game. The SALY gameplay is briefly discussed below:
1. The psychologist explains the therapy procedure.
2. Each participant must log in to start the game.
3. A main menu screen is displayed, featuring four types of mini-game engines: Simulation, Attention, TASALY, and Matching. Participants begin playing according to the guidelines and directions provided by their therapist or parent. To engage participants and enhance their learning experience, the game’s startup user interface includes an animated clip with emotional sounds, designed to capture participants’ attention and aid their learning (see Figure 10).
4. Simulation Game Engine: Figure 11 shows a screenshot of participants interacting with the simulation game. This game teaches children to recognize six basic emotions. To facilitate stimulation and emotion recognition, various 2D visual stimuli featuring different modalities are displayed randomly, one at a time, in front of the participants. As a challenge, the participant’s role is to correctly mimic the presented emotion. The simulation game tracks gameplay through metrics such as response time and the successes and failures that occur during play mode, which can be recorded and saved after each session. These recorded metrics provide valuable feedback to therapists, enabling them to evaluate clinical results following each play session.
5. Matching Game Engine: Before progressing to the next task, participants must complete a short and entertaining quiz known as the matching game after practicing with the simulation game. The matching game serves as a facial discrimination intervention aimed at helping participants identify and match emotions displayed on faces (e.g., happy, sad, etc.). A single or multiple target emotional faces are shown at the top of the screen, with corresponding “choices” displayed at the bottom. Participants are prompted to select the emotional expression that matches the target(s). For example, if the target shows a happy face, the participant must match it with another happy face. To assist players, the game starts by displaying basic emotional expressions, accompanied by sound effects. Feedback is provided through different sound effects when participants answer correctly or incorrectly. The matching game tracks gameplay using metrics such as (1) response time (the duration it takes for a participant to match a face) and (2) player scores (indicating how well the participant performed). Participants receive rewards based on their achievements upon completing the game. Two types of rewards are provided: extrinsic rewards (e.g., badges or trophies) and intrinsic rewards from playing the TASALY game.
6. TASALY Game Engine: TASALY is a fantasy, music-based social storytelling game designed to generate fun and evoke emotions among participants. It serves as an intrinsic reward intended to enhance participant engagement through two gaming mechanics: an easy-to-use graphical user interface and audio/visual feedback. The game features a self-learning component, allowing participants to retry after failures or attempt to achieve better scores. The concept for TASALY is inspired by music therapy and interactive storytelling (Marquez-Garcia et al., 2021), as music fosters emotional expression, motivation, and feedback perception. In TASALY, goals are visualized through storytelling. The game promotes cooperation, socialization, and friendship among participants while indirectly teaching them about musical scales. The narrative begins with a character receiving a message on their mobile phone, stating that it is their friend’s birthday, and that friend enjoys music. The character decides to buy a piano as a gift and visits a music shop with their friend. As a challenge within the game, the player must navigate through the musical scale, which serves as an obstacle. When the player touches any tone (e.g., Do, Re, etc.) in the musical scale, a corresponding sound effect plays, and a score is awarded. Players use the left and right arrow keys for movement and the space bar to jump. The game elements in TASALY include sound, a timer, levels, and score objects. To enhance engagement and mitigate boredom or frustration after failures, TASALY incorporates a feature known as difficulty scaling or an “even game” (Pieter et al., 2004), which adjusts the challenge level to match the player’s skill, thus increasing both the entertainment factor and gameplay strength. The storyboard illustrating the scenario scope in the TASALY game is shown in Figure 12a, while Figure 12b displays a screenshot of participants interacting with the TASALY game.
7. Attention Game Engine: The goal of the attention game is to improve the child’s attention skills and observation as it encourages the participant to observe and analyze different emotional faces. The participant is asked to memorize several images of different emotional expressions arranged in a matrix of different sizes (2×2, 3×3, etc.) from a slider. As a challenge in this game, the role of the participant is to select the right pair of similarities according to their interpretation of emotional expressions images. The difficulty levels are embedded in the matrix size and adjusted by the help of the therapist/parent. The metrics for the attention game are: (1) response time (How long in milliseconds does it take for the participant to respond?), (2) participant’s score, and (3) difficulty levels. Participants are rewarded using extrinsic reward (e.g., badges/trophies), and intrinsic reward by playing TASALY game.
8. By the end, feedback meetings were organized, and results were analyzed.
Figure 11
Figure 11. An illustration of some participants interacting with the simulation game engine.
Figure 12
Figure 12. (A) Screenshots for TASALY game storyboard. (B) An illustration of some participant interacting with the TASALY game engine.
7 Materials and methods
7.1 Selections of participants
We gathered a sample of 33 children aged between 7 and 15 years to play the SALY game for at least one and a half hours per week over a period of 7 weeks during an open trial. For each participant, data on their educational background and any special care (therapeutic or educational) was also collected. One participant was unable to continue playing because he refused to participate. Additionally, one child (3%) had fine motor skill problems. Most participants were diagnosed with autism by a psychiatrist or clinical psychologist. The sample consisted of 25 boys (76%) and eight girls (24%), referred to collectively as “Group 1.” These children were recruited from the Faculty of Graduate Studies for Childhood’s Center for Special Needs Care. This center not only provides assistance to individuals with learning difficulties but also trains parents on how to manage their child’s specific impairments using cutting-edge scientific research. To measure IQ, the Wechsler Abbreviated Scale of Intelligence (WASI) was employed. Table 3 presents the characteristics of the participants. The developed application has also been tested with educators, parents, and typically developing children to ensure its accessibility for those with Autism Spectrum Conditions (ASC). Eleven typically developing children were invited to participate; they were designated “Group 2.” This group took part in the application to evaluate its functionality and to assess the efficiency and effectiveness of its use in educational contexts.
Table 3
Table 3. Characteristics of participants.
7.2 Ethical issues
Because of the crucial importance of the research and the confidentiality considerations involved, there are a number of challenges and ethical issues that come up when people with ASC are involved in the production of an SG. A written consent form asking for permission for three separate research components was delivered to all the parents and other pertinent parties of the children participating in the study. The first was the child’s involvement in the research; the second was access to the child’s academic and medical data; and the third was keeping track of the child’s progress throughout the study.
7.3 Materials and procedures
The intervention protocol for participants consists of three distinct phases: pre-intervention, intervention, and post-intervention. After obtaining consent from both parents and children, participants and their parents took part in a two-day pre-intervention assessment. During these sessions, children underwent IQ testing that measured both verbal and performance IQ, using the 2nd edition of the Wechsler Abbreviated Scale of Intelligence (WASI-II). Meanwhile, parents completed the Social Responsiveness Scale, 2nd edition (SRS-2), and the Vineland Adaptive Behavior Scales, 2nd edition (VABS-II) to confirm the children’s diagnostic status. The WASI-II exam assesses a child’s verbal, nonverbal, and overall cognitive abilities, providing an estimate of their general intellectual capacity. The SRS-2 measures the severity of autism symptoms, evaluating areas such as social cognition, social motivation, social communication, and social awareness. The VABS-II adaptive behavior scale gauges the ability to convert cognitive potential into practical life skills, rating performance across various domains, including sociability, daily living skills, and communication.
8 Assessment, usability testing and obtained outcomes
This section offers an extensive explanation for the response to RQ3 (Table 1). It demonstrates how effectively the application functions in terms of effectiveness, enjoyment, ease of use, usefulness, satisfaction, improvements, and attitudes toward future usage from the perspectives of the participants, their parents, and other relevant stakeholders.
8.1 Usability testing
Usability testing is a technique employed to evaluate the effectiveness of the SALY game and user satisfaction. This method assesses how well-liked and efficient the system is among participants, their parents, and other relevant stakeholders. Several usability metrics, which are discussed in the following subsections, can be utilized to evaluate the system’s usability from the perspectives of both participants and their parents, as well as other stakeholders. These tests aim to gather feedback from a diverse range of users, including those without disabilities (Group 2), to ensure that individuals with Autism Spectrum Conditions (ASC) can navigate and use the system effectively.
8.1.1 Usability testing with users and their parents
To evaluate parents’ expectations and satisfaction regarding the usability of the developed application, a variety of questions were posed using a 10-point Likert scale during both pre-intervention and post-intervention assessments. The System Usability Scale (SUS) questionnaire (McLellan et al., 2011) was employed, which includes 10 questions focusing on learning efficiency, ease of learning, memorization, occurrence of execution errors, and overall satisfaction. Each question utilizes a five-point scale ranging from one (totally disagree) to five (totally agree). Sample SUS questionnaires can be found in Table 4. Participants were also invited to share their feedback on the application, including aspects they liked or disliked, as well as their favorite games. For this purpose, we used the Fun Toolkit (Gavin and Matthew, 2012), incorporating the Smileyometer rating scale to assess participants’ satisfaction. As illustrated in Figure 13, the Smileyometer is a visual analog scale that allows participants to easily express their feelings by circling one face for each question, eliminating the need for written responses.
Table 4
Table 4. Sample assessment questionnaires for parent expectations and satisfaction.
Figure 13
Figure 13. A smileyometer.
8.1.2 Usability testing with relevant stakeholders
In the realm of special educational needs, user testing alone is insufficient without expert input, as individuals with Autism Spectrum Conditions (ASC) may struggle to articulate their thoughts and may respond atypically in various assessments. To identify usability issues related to efficiency and satisfaction in the developed application, evaluations were conducted not only from the perspective of experts but also through user testing, including feedback from typically developing individuals (Group 2). This study employed a mixed-methods approach to gather insights on the app’s effectiveness and potential implementation challenges. Eight subject-matter experts were invited to participate, representing diverse academic backgrounds: two were involved in education and game design, one was a special education instructor, one was an occupational therapist, and another was a speech and language therapist. Additionally, three researchers with expertise in psychiatry and psychology, including one of the authors, contributed their insights. Experts were provided with a list of the interface and tasks associated with the four mini-game engines in the SALY gameplay for usability analysis to ensure suitability for the target audience. The experts’ perspectives were categorized into five themes: effectiveness, usefulness, enjoyment, ease of use, and attitudes toward future usage.
1. Effectiveness: Most experts agreed that the game was effective due to:
o Clear presentation of goals in the game’s introduction, which enhances children’s attention and learning (e.g., in the simulation game).
o The integration of music with visual material (e.g., in the TASALY game).
o Immediate and clear feedback provided through visual and audio elements (e.g., in the attention game).
o Defined goals and feedback that support concentration, aligning with desired learning outcomes.
2. Usefulness: Experts in psychology and psychiatry emphasized that the game is beneficial compared to traditional intervention techniques—such as social skills classes, narrative therapy, and role-playing—that can be costly, time-consuming, and tedious for participants due to repetitive exercises and long waiting lists. They noted that the game:
o Is cost-effective and supports self-confidence.
o Can aid in developing motor skills.
o Saves educators’ time.
3. Enjoyment: Experts from game design and teaching highlighted that the game is enjoyable because:
o It aims to increase concentration and curiosity among individuals with ASC.
o It incorporates audio-visual feedback and both extrinsic rewards (e.g., points, badges, leaderboards) and intrinsic rewards (e.g., from the TASALY game) to enhance motivation and engagement.
o Participants experience joy from completing tasks independently, particularly when they see their own faces on the computer screen during gameplay.
4. Ease of Use: Most experts concurred that the game is user-friendly, with simple content and a logical flow of topics. They noted:
o A good balance between challenge and skill that aligns with the player’s ability.
o Straightforward mechanics and interface that are easy to understand and access.
5. Attitudes: All experts agreed that the game could serve as an effective teaching tool for individuals with ASC in a cost-efficient manner.
Overall, the input from experts reinforced the potential of the SALY game as a valuable resource for therapeutic and educational purposes for children with ASC.
8.1.3 Obtained outcomes and the technology acceptance model (TAM)
To investigate the acceptance of the developed SALY gameplay by its target group, we employed Davis's (1989) Technology Acceptance Model (TAM). TAM is a well-established framework for predicting user acceptance of new technology, focusing on the factors that influence a user’s decision to adopt or reject it. This model provides insights into future technology usage intentions by examining five dimensions: perceived usefulness (PU), perceived ease of use (PEU), perceived enjoyment (PE), attitude toward its use (AU), and intention to use it (IU), as illustrated in Figure 14. According to Granic and Marangunic (2019) and Sprenger and Schwaninger (2021), both PU and PEU are crucial for educators and learners, significantly affecting their attitudes toward using technology in the classroom. Table 5 presents six example questions provided to the specialists in this study, with three questions assessing PU and three assessing PEU. To evaluate attitudes toward specific technologies, we utilized the number of questions indicating agreement (AgrP) and disagreement (DisP) to gauge the level of consensus on each proposition, referred to as the Degree of Agreement Proposition (DAP). Based on DAP results, Granic and Marangunic (2019) provided a table (see Table 6a) that summarizes an individual’s level of agreement or disagreement with a proposition in straightforward terms.
Table 5. Six example questionnaires for PU and PEU component assessments.
Table 6
Table 6. DAP values interpretations and obtained outcomes.
The following terms are used in the computation of DAP:
• Strongly Agree (SA)
• Partially Agree (PA)
• Neutral (N)
• Partially Disagree (PD)
• Totally Disagree (TD)
The DAP is computed using the counts of agreements (AgrP) and disagreements (DisP) based on the aforementioned terms, as follows:
Using Equation 1, Table 6b presents the survey results from five experts for each of the six sample questions, resulting in values for Disagreement (DisP), Agreement (AgrP), and Degree of Agreement Proposition (DAP). The DAP values indicate that experts strongly agree on several aspects of the application: it is perceived as helpful, easy to use, authentically developed, user-friendly, and features a well-structured flow of topics in both the Perceived Usefulness (PU) and Perceived Ease of Use (PEU) categories.
8.2 Assessment and usability metrics
The SALY system’s usability was assessed using the game data of each participant. The data for the game is stored in internal structures and contains the participant’s score for each game successfully completed, the participants’ rating based on his or her score, how long it took them to complete each activity, etc. The system creates reports in the assessment file folder when the participant has finished playing the games, providing the therapist with feedback. In order to test the system with users and to measure the efficiency and effectiveness of the application’s use in education, we considered the following usability metrics (Albert and Tullis, 2013):
1. Success score = (Number of completed tasks/Total number of attempts). The range of the value would be 0 to 1 (or 0 to 100%). It demonstrates if the requested task was successfully completed or not. Figure 15 displays the participants’ success scores for the various SALY system tasks both before and after the interventions.
2. Success rate: There are many success levels included here: a) Complete success indicates that the user completed the task correctly and without error; b) Success with a minor issue indicates that the user completed the task but encountered a minor issue; c) Success with a major issue indicates that the user completed the task but encountered a major issue; and d) Failure indicates that the user was unable to complete or finish the required task. Figure 16 sketches the participation success rate for each level of success.
3. Task time = (Time1st user + Time2 + … + Timen/Total number of users). This is how long it took the user to do the task. Task time is used to gauge efficiency. To calculate task time in terms of time-based efficiency, use Equation 2 below:
Figure 15
Figure 15. Participants’ success score.
Figure 16
Figure 16. Participants’ success rate at each level of success.
Where, T is the overall number of tasks, R is the entire number of users, Nij is the result of task i by user j; if the user succeeds in the task, Nij = 1, otherwise, Nij = 0, and Mij is the amount of time user j spent on task i. A visual representation of the time-based efficiency average rate for various SALY system task activities for diverse users is shown in Figure 17.
Figure 17
Figure 17. A diagram displaying the efficiency rate.
4. Learnability: Learnability considers both the ease of use of the task for users on their first attempt and the number of tries required to accomplish it correctly. The learnability rate for the participants’ responses in conventional and non-traditional methods is shown in Figure 18 following practice with our SALY game. It is clear that after practicing with our game, their performance improved significantly.
Figure 18
Figure 18. Learnability rate.
9 Results and discussion
This section describes the findings of the responses collected from experts, parents, and their children. This was done to help children recognize the SALY game as a learning tool and determine which features of the game are more attractive. The results are displayed as follows, and the data collected demonstrate how effective the game is:
1. Most of the participants agreed that the proposed game is helpful, straightforward to use, user-friendly, and has simple content with good topic flow.
2. Regarding game motivation, all participants enjoyed playing the game and seeing their faces on the computer screen; the favorite game was the “Attention and TASALY” which was the most selected and played the longest.
3. When it comes to game usability, most of the participants were able to play the game because participants’ preferences were achieved via factors like simple GUI, audio-visual feedback and both extrinsic (e.g., points, badges/trophies, etc.) and intrinsic rewards (e.g., playing TASALY game). Most experts rated the usability of SALY game as “excellent” based on the results collected.
4. Regarding emotion expression and emotion recognition abilities, experimental results after training show that about 83% of participants can recognize sadness, 87% happiness, 77% anger, 65% surprise, 50% disgust and 55% fear.
5. Results analysis shows that the six fundamental emotions are difficult for people with autism to recognize in general, with fear, disgust, and surprise being the hardest.
6. Our results analysis suggests that participants with ASC had an 87% achievement rate while mimic the emotional expressions of faces using a stimulus of human caricature faces and emojis in comparison with an achievement rate of 77% for human faces. These observations have been comparable to (Atherton and Cross, 2018; Rosset et al., 2008) whose studies confirmed that children with ASC prefer cartoons and items over real faces.
7. The majority of the experts, as well as parents and their children were satisfied with SALY game. Experts indicated that the system appears to have outstanding potential and can be objectively used as a teaching aid for assisting people with ASC. Parents also reported that 7 weeks of SALY’s use significantly improved their children’s performance on the different emotion recognition tasks and social skills.
1. Social Presence: Number of players, such as a single-player mode.
2. Interaction Style/Technology: How participants interact with the system via keyboard and mouse.
3. Performance Feedback: The system’s ability to convey interaction outcomes to the participant.
4. Progress Monitoring: The system’s capability to save participants’ interaction outcomes.
Table 7
Table 7. Comparison between our proposed SALY Game and the other existing frameworks.
Additionally, SALY aligns with the GDF framework (Tsikinas and Xinogalos, 2020), which emphasizes assessment as a core design element. However, Tsikinas and Xinogalos (2020) did not validate their GDF to determine the effectiveness and satisfaction level of the produced serious games (SG) for the target audience. In contrast, our framework addresses these gaps by incorporating several usability metrics, including the Technology Acceptance Model (TAM). Moreover, the SALY framework integrates design thinking methodology combined with participatory design sessions, ensuring innovative solutions to meet user needs. Several distinctions between SALY and other frameworks are also noted, as highlighted in Table 7. These include:
1. Game Portability: The system’s ability to be used in various settings such as home, hospital, or clinic.
2. Game Interface: Whether the game uses a 2D or 3D interface.
3. Adaptability: The system’s ability to adjust the game’s difficulty or challenge based on the participant’s performance.
4. Usability Testing: Verifying whether the proposed game is well-received by the target audience.
10 Conclusion
In the realm of pedagogical research, particularly within special educational needs, participatory research (PR) has garnered increasing recognition. The involvement of individuals with Autism Spectrum Condition (ASC) in developing digital technologies—where children and relevant stakeholders collaborate to devise strategies and make decisions—is becoming more prominent in the special educational needs’ community. Digital technologies, such as serious games (SGs), are often employed to help individuals with ASC learn more effectively than through conventional methods. However, engaging individuals with ASC in technology development presents challenges because each person is unique, and their needs and abilities may evolve over time. This variability complicates efforts by technology designers to interact with them effectively. Moreover, despite the existence of several frameworks, the question remains: can SGs designed for individuals with ASC benefit from their own dedicated framework? This paper addresses this question by proposing a participatory research framework that treats education as a collaborative process with goals extending beyond knowledge creation to practical application. The framework is built on two key pedagogical elements: participatory sessions and the design thinking process. It was applied in a case-based learning study to design a new SG called SALY (Simulation, Attention, Learn, and PLAY). SALY system aims to enhance attention span and emotion recognition abilities in individuals with ASC by integrating technology and learning. Three research questions were explored in this study, with data analyzed using mixed-methods research. Various usability metrics were employed to evaluate the game’s effectiveness, efficiency, and user satisfaction. SALY distinguishes itself from previous frameworks in the following ways:
1. An innovative participatory research framework was developed and implemented in the design of a new SG, called SALY; to help game developers and designers create effective SG, particularly for people with ASC. Playing the SALY game, which blends technology and learning, can help people with ASC improve their ability to recognize emotions and attention skills.
2. Participants may express their emotions through the intrinsic rewards offered by TASALY, a fantastical music-based social story game. This helps to motivate players and keeps them emotionally engaged, captivated, happy, and fully immersed in the game.
3. Through music therapy (Marquez-Garcia et al., 2021), people may communicate their feelings, which also aids in motivation and feedback.
4. SALY possesses an “even game” feature (Pieter et al., 2004), in which the game challenge level matches the skill of the human player, consequently raising the playing strength and the entertainment factor;
5. A wide range of users, including those without impairments (Group 2), were taken into account when evaluating the game.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.
Ethics statement
The studies involving humans were approved by Faculty of Postgraduate Studies for Childhood. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation in this study was provided by the participants’ legal guardians/next of kin. Written informed consent was obtained from the minor’s legal guardian or next of kin for the publication of any potentially identifiable images or data included in this article.
Author contributions
HA: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Supervision, Validation, Writing – original draft, Writing – review & editing, Data curation. MO: Formal analysis, Investigation, Methodology, Validation, Writing – original draft. HM: Data curation, Formal analysis, Validation, Writing – original draft.
Funding
The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.
Acknowledgments
The authors expressed their appreciation to the managing editor and the reviewers for their helpful comments.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Abd El-Sattar, H. K. H. (2023a). EMOCASH: an intelligent virtual-agent based multiplayer online serious game for promoting money and emotion recognition skills in Egyptian children with autism. Int. J. Advanc. Comput. Sci. Applic. 14, 1–17. doi: 10.14569/IJACSA.2023.0140414
Abd El-Sattar, H. K. H. (2023b). A new learning theory-based framework for combining flow state with game elements to promote engagement and learning in serious games. Inf. Sci. Lett. 12, 2663–2677. doi: 10.18576/isl/120660
Abd El-Sattar, H. K. H. (2024, 2024). Future metaverse-based education to promote daily living activities in learners with autism using immersive technologies. Educ. Inf. Technol., 1–38. doi: 10.1007/s10639-024-12844-3
Albert, W., and Tullis, T. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics (interactive technologies). San Mateo, CA, USA: Morgan Kaufmann.
Almeida, L. M., Silva, D. P., Theodório, D. P., Silva, W. W., Rodrigues, S. C. M., Scardovelli, T. A., et al. (2019). ALTRIRAS: a computer game for training children with autism Spectrum disorder in the recognition of basic emotions. Int. J. Comp. Games Technol. 2019, 1–16. doi: 10.1155/2019/4384896
Alves, S., Marques, A., Queirós, C., and Orvalho, V. (2013). LIFEisGAME prototype: a serious game about emotions for children with autism spectrum disorders. PsychNology J. 11, 191–211.
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Ye, D., Al-Shamma, O., et al. (2021). Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J. Big Data 8:53. doi: 10.1186/s40537-021-00444-8
Anselma, M., Chinapaw, M., and Altenburg, T. (2020). Not only adults can make good decisions, we as children can do that as well. Evaluating the process of the youth-led participatory action research ‘kids in action’. Int. J. Environ. Res. Public Health 17:625. doi: 10.3390/ijerph17020625
Atherton, G., and Cross, L. (2018). Seeing more than human: autism and anthropomorphic theory of mind. Front. Psychol. 9, 1–18. doi: 10.3389/fpsyg.2018.00528
Bai, Q., Dan, Q., Mu, Z., and Yang, M. (2019). A systematic review of emoji: current research and future perspectives. Front. Psychol. 10, 1–16. doi: 10.3389/fpsyg.2019.02221
Bakhtiar, A., Lang, M., Shelley, B., and West, M. (2023). Research with and by children: a systematic literature review. Rev. Educ. 11:e3384. doi: 10.1002/rev3.3384
Borjesson, P., Barendregt, W., Eriksson, E., and Torgersson, O. (2015). Designing technology for and with developmentally diverse children: a systematic literature review. In Proceedings of the 14th international conference on interaction design and children-IDC 15 (pp.79–88)
Bottema-Beutel, K., Kapp, S. K., Lester, J. N., Sasson, N. J., and Hand, B. N. (2021). Avoiding Ableist language: suggestions for autism researchers. Autism Adulthood. 3, 18–29. doi: 10.1089/aut.2020.0014.36601265
Conti, D., Di Nuovo, S., Buono, S., Trubia, G.Di, and Nuovo, A. (2015). Use of robotics to stimulate imitation in children with autism Spectrum disorder: a pilot study in a clinical setting. The 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Kobe, Japan, 2015, 1–6
Dunst, C. J., Prior, J., Hamby, D. W., and Trivette, C. M. (2013). Influences of a socially interactive robot on the affective behavior of young children with disabilities. Social Robots Res. Rep. 2013, 3, 1–10.
Duquette, A., Michaud, F., and Mercier, H. (2008). Exploring the use of a mobile robot as an imitation agent with children with low-functioning autism. Auton. Robot. 24, 147–157. doi: 10.1007/s10514-007-9056-5
Efilti, P., and Gelmez, K. (2023). A deep dive into the impacts of empathy on design learning and teaching. Int. J. Technol. Des. Educ. 34, 809–852. doi: 10.1007/s10798-023-09835-9
Elhaddadi, M. (2022). Technique of collecting results when using the serious game JESTIMULE dedicated to helping autistic children to recognize emotional facial expressions. J. Psych. Sci. Res. 2, 1–11. doi: 10.53902/JPSSR.2022.02.000534
Gallud, J. A., Carreño, M., Tesoriero, R., Sandoval, R., Maria, D., Israel, D., et al. (2021). Serious games to teach emotion recognition to children with autism Spectrum disorders (ASD). Acta Neuropsychol. 19, 81–92. doi: 10.5604/01.3001.0014.7569
Fabri, M., Andrews, P. C. S., and Pukki, H. K. (2016). Using design thinking to engage autistic students in participatory design of an online toolkit to help with transition into higher education. J. Assist. Technol. 10, 102–114. doi: 10.1108/JAT-02-2016-0008
Ferrari, E., Robins, B., and Dautenhahn, K. (2009). Therapeutic and educational objectives in robot assisted play for children with autism," RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication, Toyama, Japan, 2009, 108–114
Fridenson-Hayo, S., Berggren, S., Lassalle, A., Tal, S., Pigat, D., Meir-Goren, N., et al. (2017). Emotiplay: a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation. Eur. Child. Adolescent. Psychiatry 26, 979–992. doi: 10.1007/s00787-017-0968-0
Gallud, J. A., Carreno, M., Tesoriero, R., et al. (2023). Technology-enhanced and game based learning for children with special needs: a systematic mapping study. Univ. Access Inf. Soc. 22, 227–240. doi: 10.1007/s10209-021-00824-0
Gavin, D., and Matthew, H. (2012). Investigating children‘s opinions of games: fun toolkit vs. this or that. Proceeding of the 11th international Conf. on Interaction Design and Children, pp. 70–77, doi: 10.1145/2307096.2307105
Glumbic, N., Dordevic, M., and Brojcin, B. (2022). “Daily living” in Digital inclusion of individuals with autism Spectrum disorder. Autism and child psychopathology series. ed. Matson, J. L. (Cham: Springer).
Granic, A., and Marangunic, N. (2019). Technology acceptance model in educational context: a systematic literature review. Br. J. Educ. Technol. 50, 2572–2593. doi: 10.1111/bjet.12864
Grossard, C., Hun, S., Dapogny, A., Juillet, E., Hamel, F., Jean-Marie, H., et al. (2019). Teaching facial expression production in autism: the serious game JEMImE. Creat. Educ. 10, 2347–2366. doi: 10.4236/ce.2019.1011167
Habib, G., and Qureshi, S. (2022). Optimization and acceleration of convolutional neural networks. A survey. J. King Saud Univ. Comp. Inform. Sci. 34, 4244–4268. doi: 10.1016/j.jksuci.2020.10.004
Joshi, S., Verma, D. K., Saxena, G., and Paraye, A. (2019). Issues in training a convolutional neural network model for image classification. In: Proc. of Advances in Computing and Data Sciences: Third International Conf., ICACDS 2019, Springer Singapore, Ghaziabad, India, April 1046, pp. 282–293
Kay, E., Tisdall, M., Emma, C., and Lynn, J. M. (2023). Child-led research with young children: challenging the ways to do research. Soc. Sci. 13, 1–18. doi: 10.3390/socsci13010009
Kelly, S. R., Mazzone, E., Horton, M., and Read, J. C. (2006). Bluebells: a design method for child-Centred product development. ACM Int. Conf. Proceeding Series 189, 361–368. doi: 10.1145/1182475.1182513
Kelly, C., Waldron, F., and Dooley, T. (2023). “Young children as co-researchers: authentic partnership in an early childhood context” in Pushing the boundaries of human rights education: Concepts, challenges and contexts. eds. B. Mallon, F. Waldron, and C. Cassaithe, 47–64. doi: 10.4324/9781003321644-5
Key, K. D., Furr-Holden, D., Lewis, E. Y., Cunningham, R., Zimmerman, M. A., Johnson-Lawrence, V., et al. (2019). The continuum of community engagement in research: a roadmap for understanding and assessing Progress. Prog. Community Health Partnersh. 13, 427–434. doi: 10.1353/cpr.2019.0064
Kinnula, M., and Iivari, N. (2021). Manifesto for children‘s genuine participation in digital technology design and making. Int. J. Child Computer Interact. 28:100244. doi: 10.1016/j.ijcci.2020.100244
Koh, J. L., Chai, C. S., Wong, B., and Hong, H. Y. (Eds.) (2015). “Design thinking and 21st century skills” in Design thinking for education Chapter#3 (Singapore: Springer).
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., and van Knippenberg, A. (2010). Presentation and validation of the Radboud faces database. Cognit. Emot. 24, 1377–1388. doi: 10.1080/02699930903485076
Marquez-Garcia, A. V., Magnuson, J., Morris, J., and Grace, I. (2021). Music therapy in autism Spectrum disorder: a systematic review. Review. J. Autism Dev. Disord. 9, 91–107. doi: 10.1007/S40489-021-00246-X
Maun, R., Fabri, M., and Trevorrow, P. (2023). Participatory methods to engage autistic people in the Design of Digital Technology: a systematic literature review. J. Autism Dev. Disorders 54, 2960–2971. doi: 10.1007/s10803-023-06015-5
McVeety, E., and Farren, M. (2020). An action research enquiry into child voice in the primary classroom by empowering children to arrange and implement their own timetable. Educ. Action Res. 28, 383–404. doi: 10.1080/09650792.2019.1610021
Min, F., Alissa, N. A., Jianyu, F., Philippe, A., and Sheng, J. (2018). EmoStory: a game-based system supporting Children's emotional development. Extended Abstracts of the 2018 CHI conference on human factors in computing systems
de Carvalho, A. P., Braz, C. S., dos Santos, S. M., Ferreira, R. A. C., and Prates, R. O. (2023). Serious games for children with autism Spectrum disorder: a systematic literature review. Int. J. Hum. Comput. Interact. 40, 3655–3682. doi: 10.1080/10447318.2023.2194051
Pavez, R., Diaz, J., Arango-Lopez, J., Ahumada, D., Mendez-Sandoval, C., and Moreira, F. (2023). Emo-mirror: a proposal to support emotion recognition in children with autism spectrum disorders. Neural Comput. Applic. 35, 7913–7924. doi: 10.1007/s00521-021-06592-5
Pettersson, I., Lachner, F., Frison, A.-K., Riener, A., and Butz, A. (2018). “A review of method application and triangulation in user experience evaluation” in Proceedings of the 2018 CHI conference on human factors in computing systems - CHI 18 (Montreal QC, Canada: ACM Press), 1–16.
Pieter, S., Ida, S., and Eric, P. (2004). Difficulty scaling of game AI. Proceeding of the 5th International Conference on Intelligent Games and Simulation (pp.33–37)
Rafael, P. L.
, &. Nuria, M
. (2016). Using UML to model educational games. 8th International. Conference on Games and Virtual Worlds for Serious Applications (VS-Games), pp.1–4
Robins, B., Dautenhahn, K., Billard, A., and Te Boekhorst, R. (2004). Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills? Universal Access Inform. Soc. 4, 105–120. doi: 10.1007/s10209-005-0116-3
Rosset, D. B., Rondan, C., Da Fonseca, D., Santos, A., Brigitte, A., and Christine, D. (2008). Typical emotion processing for cartoon but not for real faces in children with autistic Spectrum disorders. J. Autism Dev. Disord. 38, 919–925. doi: 10.1007/s10803-007-0465-2
Samar, A., Abd El-Sattar, H. K. H., Abdel-Rahman, M. H., and Ghaleb, F. (2023a). COVID-19 infection segmentation using hybrid deep learning and image processing techniques. Sci. Rep. 14, 1–17. doi: 10.1038/s41598-024-53425-1
Samar, A., Abd El-Sattar, H. K. H., Abdel-Rahman, M. H., and Ghaleb, F. (2023b). Robust object recognition with deep learning on a variety of datasets. Int. J. Intelligent Engin. Syst. 16, 436–449. doi: 10.22266/ijies2023.0831.35
Scott-Barrett, J., Cebula, K., and Florian, L. (2023). The experiences and views of autistic children participating in multimodal view-seeking research. Int. J. Res. Method Educ. 46, 342–373. doi: 10.1080/1743727X.2022.2149728
Serret, S., Hun, S., Iakimova, G., Lozada, J., Anastassova, M., Santos, A., et al. (2014). Facing the challenge of teaching emotions to individuals with low-and high-functioning autism using a new serious game: a pilot study. Mol. Autism. 5, 1–17. doi: 10.1186/2040-2392-5-37
Simpson, K., Imms, C., and Keen, D. (2022). The experience of participation: eliciting the views of children on the autism spectrum. Disabil. Rehabil. 44, 1700–1708. doi: 10.1080/09638288.2021.1903100
Sprenger, D. A., and Schwaninger, A. (2021). Technology acceptance of four digital learning technologies (classroom response system, classroom chat, e-lectures, and mobile virtual reality) after three months’ usage. Int. J. Educ. Tech. Higher Educ. 18:8. doi: 10.1186/s41239-021-00243-4
Tang, Y. M., Chau, K. Y., Kwok, A. P. K., Zhu, T., and Ma, X. (2022). A systematic review of immersive technology applications for medical practice and education - trends, application areas, recipients, teaching contents, evaluation methods, and performance. Educ. Res. Rev. 35:100429. doi: 10.1016/j.edurev.2021.100429
Tene, T., Marcatoma Tixi, J. A., Palacios Robalino, M. L., Mendoza Salazar, M. J., Vacacela Gomez, C., and Bellucci, S. (2024). Integrating immersive technologies with STEM education: a systematic review. Front. Educ. 9:1410163. doi: 10.3389/feduc.2024.1410163
Tsikinas, S., and Xinogalos, S. (2020). Towards a serious games design framework for people with intellectual disability or autism spectrum disorder. Educ. Inf. Technol. 25, 3405–3423. doi: 10.1007/s10639-020-10124-4
Vaughn, L. M., and Jacquez, F. (2020). Participatory research methods – choice points in the research process. J. Participatory Res. Methods 1, 1–13. doi: 10.35844/001c.13244
Vaughn, L. M., Whetstone, C., Boards, A., Busch, M. D., Magnusson, M., and Määttä, S. (2018). Partnering with insiders: a review of peer models across community-engaged research, education and social care. Health Soc. Care Community 26, 769–786. doi: 10.1111/hsc.12562
Wohofsky, L., Marzi, A., Bettarello, F., Zaniboni, L., Lattacher, S. L., Limoncin, P., et al. (2023). Requirements of a supportive environment for people on the autism Spectrum: a human-centered design story. Appl. Sci. 13:1899. doi: 10.3390/app13031899
Citation: Abd El-Sattar HKH, Omar M and Mohamady H (2024) Developing a participatory research framework through serious games to promote learning for children with autism. Front. Educ. 9:1453327. doi: 10.3389/feduc.2024.1453327
Received: 22 June 2024; Accepted: 22 October 2024; Published: 25 November 2024.
Disclaimer:
All claims expressed in this article are solely those of the authors and
do not necessarily represent those of their affiliated organizations, or
those of the publisher, the editors and the reviewers. Any product that
may be evaluated in this article or claim that may be made by its
manufacturer is not guaranteed or endorsed by the publisher.
Hussein Karam Hussein Abd El-Sattar
Manal Omar2
