Assessment of the Effects of Digital Educational Material on Executive Function Performance

The use of digital educational material in classrooms has increased in recent years, now constituting a major market for publishers and companies. The present research addresses a digital material assessment in two different areas. First, the study through an evaluation of internal data collected on the platform shows an analysis of the use and implementation of digital material into the school. Secondly, it exposes an external examination of the effect of using digital material at school on students’ executive functions. The sample consisted of 168 children aged 8–11 from a state school in the Community of Madrid (Spain). The instrument selected for the external assessment of academic progress was the Evaluación Neuropsicológica de las Funciones Ejecutivas en Niños (ENFEN) [Neuropsychological Assessment of Executive Functions in Children]. The Learning Analytics system of the Smile and Learn platform recorded the students’ use during class. According to the usage analysis, the results obtained show preference of using activities from Logic and Spatial worlds. In the external analysis of the effect of the learning material, the results record a significant effect using activities in Logic and Spatial worlds with the Gray Trails task, which involves spatial perception, processing speed, and working memory, among others. A second analysis to contrast the results with a post hoc design approaches relationships among executive functions as involved in tasks like Gray Trails, Interference, and Ring Tasks within the usage of Spatial and Logic activities. The need for further research to improve these materials for enhanced learning and the extrapolation of training from executive functions to other tasks is discussed. Likewise, limitations of the implementation and design of these materials are pointed out.

Introduction

Since the emergence of the first technologies in the 80’s, such as videos and computers, the aim of using these resources, as of how they can help learning within the educational sphere, has been a subject of study. This involved a stream of research that has focused on the use of digital resources for teaching students, with examples on the use of videos (e.g., Zhang et al., 2006) and the implementation of digital resources for computers in classrooms (Santiago et al., 2009; Lopez-Rosenfeld et al., 2013; Goldin et al., 2014; Mateos et al., 2016). Today, with the increase in the number and use of electronic devices (computers, smartphones, and tablets), the market for digital educational material is constantly growing (Blumberg and Fisch, 2013; Hirsh-Pasek et al., 2015; Hickmott et al., 2016). This growth has led to a greater use of technologies in the classroom (Lieberman et al., 2009). Nevertheless, researchers and professionals within the educational field are still seeking to understand how this material helps students to learn.

If the market tends toward the use of these digital materials as a classroom resource, then internal and external assessment strategies need be designed and put in place to help foster the students’ learning process (Hirsh-Pasek et al., 2015; Mateos et al., 2016). Given the huge volume of digital material that is appearing on the market, it is often difficult to assess its use and usefulness. This point constitutes a weakness of the material being used in the educational system (Lieberman et al., 2009; Hirsh-Pasek et al., 2015).

As first goal, this paper considers an assessment of the real use and implementation of digital material by teachers into a school. This objective pursues to describe which areas used more digital material as support to their lessons. This may reveal preferences for different areas of knowledge or subjects at school for using activities. The educational platform Smile and Learn has the advantage of including activities for all the subjects to observe this kind of users’ behavior with freedom of choice. In second place, this study analyzes the effect of using digital material at class in pupils’ executive functions. The executive functions are analyzed as a variable to measure the academic performance (C.f. Best et al., 2011). This goal contributes to clarify the implications that the use of digital resources as support for lessons may have in learning. Moreover, both aims may approach new directions to designing and developing digital material based on users’ experience. The second goal included a post hoc experiment with the objective of providing more information about the relationships among digital material usage and the scores obtained at different tasks by the ENFEN battery.

Present Study

This study is structured with a brief definition of the platform Smile and Learn in Section “Platform Smile and Learn.” Section “Digital material design” is based on game design, followed by the digital material used in this study and an introduction to this field. Section “Theoretical background: Digital material for learning” provides a theoretical background to the use of digital material for learning, as well as its relationship to executive functions. Furthermore, an explanation of the relationship of executive functions to learning according to the review of the relevant literature for this study is included. In Section 4, materials and methods are described. Results are developed in section “Theoretical background: Digital material for learning.” First, the use of the platform is analyzed. Then the section is organized presenting the results of the effect of the use of the digital material on the executive functions. These results are followed by the analysis of a study designed to contrast the time of use with the effects obtained in the previous analysis. The discussion and limitations of the study are presented in section “Discussion,” followed by the conclusions (see section “Discussion”) and the implications of this study (section “Implications of the study”).

Platform Smile and Learn

Smile and Learn is an intelligent platform in the field of learning technologies available for the most common operating systems (Android, iOS, Linux, and Windows). When loaded, the application displays an environment with 8 different topic worlds, according with the knowledge they work (classified in Science, Spatial, Logic, Literacy, Emotions, Arts, Multiplayer, and one additional world consisting in a virtual city). These areas correspond to the various subjects on the school curriculum, providing teachers with Supplementary Material and activities to be used in each classroom topic (Lara Nieto-Márquez et al., 2020). As of February 2020, the platform contains more than 5000 activities organized in games, quizzes, videos, and tales (Figure 1). Most of the activities can be played in different levels of difficulty.

FIGURE 1

Figure 1 | Screenshot of Smile and Learn’s main screen.

Activities share some characteristics and game elements in common for teaching schools’ concepts included in the curriculum area. Moreover, these activities share the possibility of training executive functions or cognitive skills as reasoning, attention span, decision making, monitoring, working memory, language, or logical thinking. Some games, especially within the worlds of Science, Logic, and Spatial, are designed with the purpose of working problem-solving, planning or processing speed as well. These activities featured the characteristics of the digital material to work executive functions through active learning methodologies (Cf. additional material for further information).

Digital Material Design

The design of digital educational material is an interdisciplinary area. In this sense, one can find guides or theories from the area of psychology or pedagogy for the elaboration of these materials. Within these disciplines, there are sundry pedagogical models for the purpose of being adapted, or used to structure, the design and development of digital material, as well as for explaining how students’ learning benefits by the use of digital activities. Some of the educational theories that apply to digital material include the Cognitive Theory of Multimedia Learning, based on cognitive sciences (Mayer, 2005), which focuses on the different channels through which information is processed for learning. Other theories of learning and pedagogical models center on cognitivism, behaviorism, and constructivism (e.g., Siang and Rao, 2003; Bellotti et al., 2011; Arnab et al., 2015; Lim et al., 2015; Mateos et al., 2016; Arnab and Clarke, 2017). There are also models based on theories of intelligence (Sajjadi et al., 2016; Pérez et al., 2018). These models work by establishing a link among the different multiple intelligences (as proposed by Gardner, 1983), with the use of technology.

At the present time, a combination of several of said educational theories is required to explain how the game elements, the context, and the design of tasks may influence the learning process (Lim et al., 2015; Patino et al., 2016; Slussareff et al., 2016). Some examples of game elements include feedbacks of right and wrong answers, time progress, levels, rewards, and tokens. Game elements need to join learning theories to clarify their educational purposes. In the same way, game elements are going to provide the characteristics of meaning and motivation being critical for enhanced learning. Students’ motivation, attention, and engagement are achieved through different professional profiles, such as game designers for applying gaming features (gamification), to digital activities or creating serious games (Abdelali et al., 2016). These gaming features, which may be classified as mechanical, dynamic, and aesthetic (Hunicke et al., 2004), can be related to the tenets of behavioral psychology, cognition and learning theories (Tan et al., 2007; Lim et al., 2015). In turn, students’ cognitive development and their age influence learning acquired through the use of technology, constituting a factor to be taken into consideration by designers (Tan et al., 2007; Revelle, 2013). In order to properly apply these concepts to the preparation of material, it is important to understand the link between gaming features and learning theories, which would provide a more holistic view of how learning takes place. This knowledge would facilitate the use of digital material as a resource, helping teachers in their daily routine (Arnab et al., 2012; Lim et al., 2015; Abdelali et al., 2016). The challenge in the creation of digital materials arises from including game elements following solely coding and design patterns, not taking into account the integration of more abstract concepts within the overall learning process (Cf. Arnab and Clarke, 2017).

This imposes a limitation at an educational level for designing goals, as the absence of defined guidelines for the creation of games means that design methods and assessment procedures come diffuse in their application (Slussareff et al., 2016). The same thing happens the other way around, finding that certain mechanics, contexts, and components of digital activities could be explained by several educational theories. Cognitivism and constructivism are the more prominent theories when formulating design and assessment methods, not ignoring the superimposition of learning theories on activities or games on digital media (Siang and Rao, 2003; Lim et al., 2015; Mateos et al., 2016; Patino et al., 2016). This entails a complex process for establishing a guide for better producing digital activities for learning (games, videos, interactive stories, etc.), hindering assessment and study replication (Arnab and Clarke, 2017).

Besides the activities’ pedagogical requirements, there is a need to complement them with other fields, as of developers, human computer interaction (HCI), user experience (UX), user experience research (UXR), cognitive ergonomics, and information technology (IT), among others. These profiles outline the need to analyze and explore the interactions between the user and the digital material depending on their actions and decisions, as well as the relationships between the game’s components and educational norms (Tan et al., 2007; Revelle, 2013; Arnab et al., 2015; Abdelali et al., 2016). The diversity of interactions and feedback available in different types of devices (computers, tablets, or smartphones) has the potential to support learning if their design is consistent. Accordingly, several guides (e.g., Hunicke et al., 2004; Tan et al., 2007; Bellotti et al., 2011; Arnab et al., 2012, 2015; Lopez-Rosenfeld et al., 2013; Lim et al., 2015; Abdelali et al., 2016; Sajjadi et al., 2016; Arnab and Clarke, 2017) have been drawn up with proposals on the design, development, and assessment of certain materials such as games. These guides seek to connect the different tasks involved in the production of digital material for learning, or the mechanics of the games to teaching, and pedagogical models. Thus, the aim of such framework guides is to balance the different fields involved in creating digital material.

Furthermore, there are regulations established by organisms for the development of quality digital material, such as the standard UNE 71362:2017 (AENOR, 2017). Nevertheless, these documents often fail to specify limitations when implementing their design and development process. To date, there is no generally accepted guide or framework validated for the design of digital educational material and its assessment. This is probably due to the difficulty of implementing the variety of digital materials, along with the breadth of the educational areas, ages, teaching staff, and, in general, the educational community. The different frameworks that are being drawn up for the design of games reinforce the intention to improve the material, but it still needs to be assessed regarding students’ functional learning progress and according to the settings.

Theoretical Background: Digital Material for Learning

Classroom resources involving digital material could help students in terms of motivation and their attention span, and as a means for galvanizing them toward specific learning goals (Arnab et al., 2012; Hirsh-Pasek et al., 2015; Mateos et al., 2016). Moreover, many of the digital tools used provide feedback and instructions that help students with their progress and their adjustment to the learning process (Bellotti et al., 2011). A further advantage comes with the possibility of customizing activities, increasing the difficulty and raising levels, as well as the repetition of activities until the learning goals are achieved, which is vital for improving the students’ executive functions (Lieberman et al., 2009; Diamond, 2012; Blumberg and Fisch, 2013).

There are several studies dealing with the relationships between learning and enhanced executive functions and skills through the use of digital tools and materials (e.g., Santiago et al., 2009; Blumberg and Fisch, 2013; Oei and Patterson, 2013; Castellar et al., 2015). Executive functions are developed during infancy and maturity, attaining their full during early adulthood (Davidson et al., 2006; Best et al., 2009; Karbach and Unger, 2014). References are thus to be found indicating that the use of technology may improve or further the development of cognitive skills and learning (Blumberg and Fisch, 2013) in areas as visuospatial skills, problem-solving, logical thinking, or working memory, as in the study of Castellar et al. (2015). Other studies in the field of digital material usage, as Mackey et al. (2011), show improvement in reasoning or speed training, or Diamond (2012), in reasoning and working memory. This type of studies on the training of executive functions among school-aged children may therefore play a part in fostering the development of students’ skills, their instruction, and their ability to transfer knowledge. Sundry studies, such as those by Bull and Scerif (2001), Gathercole et al. (2004), Best et al. (2011), Diamond (2012, 2013), and Goldin et al. (2014), highlight executive functions as predictor variables of academic performance. Homer et al. (2018) also point out the possibility that the use of digital materials with game or video game elements can be cognitive skills training tools. Different scholars (e.g., Gathercole et al., 2004; Cheung and Slavin, 2012; Goldin et al., 2014; Pérez et al., 2018) have highlighted possible improvements in the different areas of academic knowledge through the development of skills involving abstract thinking, planning and observation (Lieberman et al., 2009; Titz and Karbach, 2014).

As already stressed, many of the applications available on the market today may help in the training and improvement of executive functions, as well as in the possibility of extrapolating the skills acquired to other tasks (Jaeggi et al., 2011; Lopez-Rosenfeld et al., 2013; Oei and Patterson, 2013; Hirsh-Pasek et al., 2015; Hickmott et al., 2016). Nevertheless, for this to happen, the instructions and design of the digital materials (games, videos, activities, etc.) need to include the students’ requirements in terms of cognitive and emotional development (Tan et al., 2007; Diamond, 2012). Likewise, these educational resources should be produced according to a pedagogical foundation defined by experts that understand the students’ needs and skills with a view to acquiring or training the learning process, as well as to prompt beneficial behavioral changes (Lieberman et al., 2009; Arnab et al., 2012; Arnab and Clarke, 2017). Tasks to assess executive functions used to be multi-domain or complex, encompassing thus the assessment of several functions (Bavelier et al., 2012; Karbach and Unger, 2014; Titz and Karbach, 2014). For example, working memory leads to the development of processing, storage, and reasoning skills. It also helps in the selection of information received by different channels (auditory and visual) required for undertaking a task, as is mentioned by the Cognitive Theory of Multimedia Learning (Mayer, 2005). Working memory can be related to calculus by assisting in the processing of abstract concepts, relating in turn to visuospatial and verbal skills. Thus, visuospatial and verbal skills are involved, as also being neuroanatomically and functionally related, in the learning of mathematical concepts, the reasoning of operations, and the implementation of what has been learnt (Klingberg, 2010; Titz and Karbach, 2014). Therefore, in this study, executive functions will be used as a predictor of students’ academic progress, following the line of study as Best et al. (2011).

Thus, executive functions can be defined as the cognitive processes that can support students’ academic success. That is, executive functions are skills on which student learning can be based or promoted to achieve academic goals (at this point in the achievement of goals, academic objectives converge with the objectives of educational game design). Studies such as Rutherford et al. (2018) support this relationship between executive functions and student learning. More specifically, it is focused on students’ self-regulated learning to achieve their academic goals in different areas of knowledge. Nonetheless, there is still a need to continue investigating the transfer of these skills to the academic field according to the training formats considered, multi-domain, or complex, which encompass the use of several functions, or simple ones in which the highlight is the training or assessment of a single function (Bavelier et al., 2012; Karbach and Unger, 2014; Titz and Karbach, 2014). There are several models that involve a description of the structure and workings of executive functions (e.g., Miyake et al., 2000; Lehto et al., 2003; Diamond, 2013). Among the executive functions in studies of children and learning, the following stand out: inhibition, working memory and cognitive flexibility, and control of interference (e.g., Brocki and Bohlin, 2004; Best et al., 2009; Diamond, 2013). This may be due to established frameworks such as that of Miyake et al. (2000) in which three separated, but related, components are defined: shifting, inhibition, and updating.

On the other hand, in school tasks or games this separation for training of executive functions in the classroom is complex. School tasks focused on learning usually require the use of multiple cognitive processes, which would include executive functions (e.g., Titz and Karbach, 2014). On the other hand, the possibility of training executive functions according to students’ cognitive skills with digital materials may help them to regulate their learning and study strategies. Thus, it is related to learning in studies such as Bull and Scerif (2001) with the focus on the mathematical area, which points to the importance of working the flexibility and ability to inhibit learning strategies to apply other strategies learned or more useful. Bull and Scerif (2001) highlight the importance of developing multiple study strategies to achieve significant learning in educational environments. Studies such as Best et al. (2011), Gathercole et al. (2004), or Meixner et al. (2019) relate these processes to academic performance in different areas of knowledge such as language, mathematics, science, or reading and to skill development. Thus, digital activities may include play elements that help students work on a variety of tasks that involve active participation (Best et al., 2009; Blumberg and Fisch, 2013). As regards tasks, changes following the training may be due to the number of repetitions or tests of the task, the feedback the student receives, or the adjustment of the levels of difficulty prompted by the progress made (Best et al., 2009; Klingberg, 2010; Bavelier et al., 2012). However, limitations are also found in the literature when analyzing executive functions because many times multiple executive functions are involved in an academic task. There is still a need to continue investigating detailed transference of said skills into the academic fields according to the training formats considered. Interventions to analyze the transfer of executive functions to different activities often require long times and resources (Martins and Gotuzo, 2017; Sulik et al., 2020).

In the literature review gathered of neuropsychological tests to evaluate executive functions, we find different examples that are presented below. It should be noted that neuropsychological tests are criticized for their impurity when evaluating executive functions (Brocki and Bohlin, 2004; García et al., 2014; Howieson, 2019). Thus, we can find tests such as Rinat et al. (2005), to assess verbal fluency; the Trail Making Test (Bowie and Harvey, 2006); maze construction types of test as Porteus (1933); Go/No go task (Simmonds et al., 2008; Redick et al., 2011); Stroop test (Zysset et al., 2001; Homack and Riccio, 2004) or Tower of Hanoi (Bull et al., 2004). Portellano et al. (2011) rely on these tests in the development of their 6 tasks for assessing executive functions and student performance in elementary education. As mentioned before, these tasks are not specialized or structured to evaluate executive functions independently. These tasks cover several variables that can affect student performance that are not captured in these measurement instruments (Howieson, 2019). On the other hand, they have been used in the educational research as performance test purposes (Anderson, 2001; García et al., 2014).

In this way, the present study aims to contribute to the study of the implementation of digital material and its use in primary school. Likewise, a general evaluation of the effect of the use of the digital educational material designed by Smile and Learn on the students’ executive functions is carried out. This evaluation focuses on the effect of different digital activities, organized in worlds of the Smile and Learn platform according to their characteristics, with the different evaluation tasks of ENFEN (Portellano et al., 2011). Thus, this external evaluation wants to check the transference to other tasks, extending the use of collected data not only to the platform environment. There are scarce studies of the effect of the use of digital materials on primary school students, which makes it difficult to carry out an external evaluation with a specific test that evaluates processes that occur during activities or games. Therefore, a test that is shown as a game for children, in these students’ mother tongue (Spanish), was used. The approach of Portellano et al. (2011) is followed in the analysis of the results and discussion. This evaluation of digital material will provide future design and improvement approaches for digital material as well as for the implementation of these resources in schools.

Materials and Methods

Participants

The study involved a sample of 168 students. This is a convenience selection undertaking non-probability sampling. It was performed at a state school in the Community of Madrid selected in a pilot group composed by 49 state schools for implementing the Smile and Learn Platform. The criteria for selecting the sample were the number of electronic devices available at school for the use of the materials, availability of teachers to receive training, and interest of the educational community in the study. The state school selected has a medium socioeconomic level. The representability of this school is 0.12% among all states’ schools at the Community of Madrid. Moreover, the percentage of representability is 0.07% of primary education students at the Community of Madrid and 0.09% in Madrid city.

The students’ ages ranged between 8 and 11 (M = 8.71; SD = 0.091), enrolled in the 2018–2019 school year, studying their 3rd, 4th, and 5th years of primary education. From these 168 students, 74 are in 3rd of primary (44.05%), 46 are in the 4th year (27.38%), and 48 in the 5th (28.57%). There are 90 boys (53.57%) and 78 girls (46.43%).

Instruments

ENFEN

The external assessment of executive functions involved the use of the task’s battery ENFEN [Neuropsychological Assessment of Executive Functions in Children] validated at the Community of Madrid by Portellano et al. (2011). This instrument pursues to approach an external assessment about student progress with digital material usage. ENFEN enables us to evaluate students’ progress in essential processes where executive functions take part. Before completing each activity, the ENFEN battery promotes the students’ training prior to its undertaking. This questionnaire is provided by Editorial TEA, including all the materials for measuring students’ executive functions (Cf. Supplementary Material). It consists of six different tasks to assess multiple cognitive processes in relation to executive functions grouped in four areas:

• Verbal Fluency Test. This area is subdivided into two tasks:

- Phonological Fluency (F1): The words in the phonological fluency test have to begin with a specific letter. All the existing words the student recalls beginning with the right letter in 1 min are considered right answers. This task involves executive functions as verbal memory, expressive language, and breadth of vocabulary.

- Semantic Fluency (F2). This task requires providing all the possible words in a chosen category. It is considered for final score the right words or names included within the said category that the student recalls in 1 min. This task works semantic memory and vocabulary breadth.

• Trails. This area is also subdivided into another two tasks:

- Gray Trails (S1) The task involves drawing a line to connect numbers as quick as possible, without making a mistake or avoiding a number. In Gray Trails, the numbers to be connected go in descending order. This task involves spatial perception, processing speed, logical reasoning, working memory, visual-motor efficiency, anticipation, and foresight.

- Color Trails (S2). The task involves drawing a line to connect numbers as quick as possible, without making a mistake or avoiding a number. In this case, a color code is added to the numbers to be alternated in the links. The numbers to be connected go in ascending order. This task assesses following executive functions: inhibition capacity, impulsivity control, sustained attention, resistance to interference, spatial perception, processing speed, logical reasoning, working memory, visual-motor efficiency, anticipation, and forecasting capacity.

For both tasks, the scores are calculated in terms of right and wrong answers and omissions, as well as the time taken to complete the task. The final score of each task used in this study corresponds with right answers minus omissions and wrong answers. This number is divided by the time in seconds that students need to complete the task and multiplied by one hundred.

• Rings task (A). This task involves drawing the example figures the students are shown in a notebook. The figures are made in a three-column milieu, moving different color rings among them to copy the image on the notebook. The test’s score is the sum of the time the child has taken to complete each one of the figures. As the score is a time-based measurement, it is inversely proportional to the improvement in the scores. Scores used at this study are the summation of the time in seconds that students need to complete the fourteen figures. This task assesses the following executive functions: inhibition capacity, impulsivity control, sustained attention, resistance to interference, spatial perception, processing speed, logical reasoning, working memory, visual-motor efficiency, anticipation, and forecasting capacity.

• Interference (IN). The interference test is based on Stroop tasks. Each student is required to say aloud the 39 word’s ink color without reading out the word actually spelt out. The final score used in this study is calculated according to the number of right answers with points deducted for substitutions or omissions. This number is divided by the time in seconds taken to complete the task and multiplied by one hundred. This task assesses the following executive functions: inhibition capacity, impulsivity control, sustained attention, resistance to interference, spatial perception, processing speed, logical reasoning, working memory, visual-motor efficiency, anticipation, and forecasting capacity.

Learning Analytics Systems at Smile and Learn Platform

The Smile and Learn platform was used as the instrument for registering and providing metrics upon the time of usage by students in each world. This is an internal data collection to cross with the external assessment in order to clarify effects and relations between variables measured. Every time an activity is played by a child, the duration and some additional feedbacks are recorded, allowing the company to analyze the interaction of a child. In this study, “time of usage” corresponds with the time span from beginning to end of the activity included within each world. Time of usage was converted to be measured in minutes.

Procedure

Prior to experimentation, information was provided about the procedure, being followed by consent obtained from the school community, specifying that the data collected would be used solely for research purposes involving the project in question. The assessment of the digital material developed involved a quasi-experimental study during the school timetable.

Training of teachers was also held prior to the implementation of the intelligent platform. These sessions were used to explain the teachers how the platform worked, along with the use of some of its features, as customization, and the arrangement of sessions for organizing activities as a task, etc. In order to choose the digital material that could be used by each grade, a guide for teachers was created for linking the platform’s activities with school curricula. The use of teaching activities (games, quizzes, videos, and tales) included in each world has been undertaken freely. Once the Platform is installed on the devices, the students did not need adult supervision to perform the activities. The management of the content to work can be done by adults through the management web, which is independent of the platform.

The ENFEN test battery was carried out individually on each of the students by two evaluators from outside the center, in specially designed areas of the school environment. The assessors were trained in the application of the tests before their use with the students. The initial data gathering or pretest with the ENFEN battery took place in October, at the start of the 2018–2019 school year. This was followed by the implementation of the Smile and Learn intelligent platform in classrooms over a 6-month period. During the 6 months of the use of the activities, contacts with the teachers were continued in order to know the limitations they had and to help them in the use of the platform. At the end of the 2018–2019 school year, by May, posttest data were collected.

Support was provided during the use of the platform, where some specifiers can be outlined: for instance, teachers between 1st and 2nd cycle vary. On the other hand, there is much more material on the platform designed at curriculum level for 3rd and 4th grades than for 5th grade.

Data Analysis

The software used to analyze data was SPSS Statistics 22. The children’s user data collected by the platform have been used to map the student’s progression. The data collected have been used to track these users to obtain a ratio of 1:1 between the user of the platform and in the classroom. Students in the database were codified to combine data from the platform (time of usage) and the scores taken by the ENFEN battery. Data were organized by each student to allow SPSS Statistics identify single cases that follow the conditions for statistical methods. The time of use has been calculated for each one of the worlds based on an aggregate of the times of use of the component activities. Thus, in order to answer the different goals of the research, two different studies were carried out.

For answering the first goal of analyzing the use of digital materials at schools, we use descriptive statistics. Based on descriptive analysis, we made a selection of those worlds more used. Then, an ANOVA is made for each of the worlds to provide a more in-depth description of the use of digital material. The differences between the courses for each of the worlds that group the activities in different areas are analyzed by Bonferroni.

The scores on ENFEN tasks by course are also analyzed to see what effect age might have. A MANOVA is done, and the courses are compared using Bonferroni.

In order to analyze the second objective, which purposes to observe whether there is an effect on executive functions given the usage of digital activities, for this, an ANOVA of repeated measures, followed by a linear regression analysis, was applied to determine the relation of such effects. Dependent variables were the scores obtained via ENFEN with pretest and posttest. The independent variable between subjects is students’ grade. Time of usage by different worlds (Logic, Spatial, Science, and Arts) was included as covariable. For the linear regression, the difference between the scores obtained in the posttest vs. the pretest was calculated and such difference was used as a dependent variable. Both the time of usage of each world and the grade were used as independent variables.

The hypothesis for this second goal is defined by each task as follows:

H0: The use of digital educational material classified by worlds in the platform (Logic, Space, Science, and Arts) has no effect on tasks involving executive functions.

H1: The use of digital educational material classified by worlds in the platform (Logic, Space, Science, and Arts) has an effect on tasks involving executive functions.

Then, in order to contrast and compare the results obtained from the second objective, a post hoc experimental design was carried out. By means of this design, two groups were created according to the use of the digital materials, analyzing the relationship of the time of usage of the digital activities with the scores obtained. For the creation of the groups, the time of use variable was dichotomized by the average in each of the worlds (Science, Logic, Spatial, Arts). Group 0 corresponds to the students who had a lower use of the platform (corresponding to a control group), and group 1, showing a higher time of use. Dependent variables are the scores obtained via ENFEN, and the independent variable is the time of use of each of the worlds. A general linear model (GLM) was chosen for statistical analysis. In this way, we want to respond to the relationship between usage vs. scores obtained. For this analysis, hypothesis is defined by each battery task as follows:

H0: There is no relationship between longer use of applications (classified by worlds in the platform) and scores taken on the battery task.

H1: There is a relationship between increased time of use of applications (classified by worlds in the platform) and scores taken on the battery task.

Results

Analysis of Usage and Task Scores

Overview Analysis

First, the students’ general use of the different activities was analyzed to find out which worlds had been most used. The children’s general use of educational activities according to their grouping by worlds in descending order is as follows: Logic (86.90%), Spatial (79.76%), Arts (27.38%), Science (20.83%), Literacy (14.88%), Multiplayer (11.31%), and Emotions (7.14%). Participants made more usage of those applications related to logic and spatial worlds than those related to all the other worlds. More information of usages averages by worlds is included in Figure 2. Descriptive statistics of the worlds showing usage higher than 20% are represented.

FIGURE 2

Figure 2 | Graphic of general descriptive statistics of the time of usage in Science, Logic, Spatial, and Arts.

The analysis of the descriptive statistics for the scores obtained in the pretest and posttest was also carried out. Results of student’s scores via the ENFEN battery taken at pretest and posttest are provided in Table 1. A light improvement in the results of the test of Phonological fluency (F1), Semantic fluency (F2), Gray Trails (S1), Color Trails (S2), Rings task (A), and Interference (IN) can be observed.

TABLE 1

Table 1 | Descriptive statistics of children’s scores by groups from the ENFEN battery.

Analyses by Class Groups

In order to study in depth the use of the differences by grade, the world usage by school years in the 3rd, 4th, and 5th years of primary education was analyzed through descriptive statistics. The results obtained from the descriptive statistics by grade and worlds usage are shown in Figure 3.

FIGURE 3

Figure 3 | Graphic of descriptive statistical by grades of time of use in Science, Logic, Spatial, and Arts.

Figure 3 shows the distribution of usage by grade according to the time of use in each world. To observe the difference in use between the activities and the different academic courses, an ANOVA was carried out. Significant differences are found in the use of activities classified in the worlds of Logic F(2) = 12.499, p < 0.001; Spatial F(2) = 13.460, p < 0.001; and Arts F(2) = 4.229, p = 0.017 (p < 0.05). In exploring the differences in the use of material between the different academic courses, the comparison of groups was analyzed using Bonferroni. No significant differences were found with the use of the Science activities and the academic courses. Thus, significant differences are obtained in usage in Logic, among the groups of 3rd with 4th (p < 0.001) and 5th (p = 0.026, p < 0.05); in Spatial, among 4th with 3rd (p < 0.001) and 5th (p < 0.001) and in Arts, showing 4th with 5th (p = 0.023, p < 0.05). These differences in use among 3rd with 4th and 5th grades may be due to the teachers’ preferences and performance, or the amount of content delivered for each of the grades.

Results from students’ scores by grades obtained via the ENFEN battery, taken at pretest and posttest, are provided in Table 1. In this table, the general descriptive statistics of the scores taken by grades can be found. In order to know the initial point of students and the differences that could be found between the scores obtained in each of the courses, a MANOVA was carried out. In the analysis of the results of the pretest by course, the following data are obtained for the course variables and scores of the ENFEN tasks: Grade F(1) = 5,057, p < 0.001; F1 F(2) = 2.072, p = 0.129 (p > 0.05); F2 F(2) = 7.024, p < 0.001; S1 F(2) = 8.894, p < 0.001; S2 F(2) = 20.519, p < 0.001; A F(2) = 8.274, p < 0.001; and IN F(2) = 12.176, p < 0.001. Significance is observed in the scores obtained in ENFEN between courses. This may be the effect of the cognitive development of each of the students depending on the grade they are in. Further in the study, we analyzed the differences that could be found by the academic courses of 3rd, 4th, and 5th grades of elementary school by Bonferroni.

Thus, in the pretest differences show among the scores obtained in the grades of 5th with 3rd (p < 0.001) and 4th (p = 0.026, p < 0.05) in Semantic fluency (F2), of 3rd with 5th in Gray Trails (S1) (p < 0.001), Color Trails (S2) (p < 0.001); and Interference (IN) (p < 0.001); and of 3rd with 4th in Color Trails (S2) (p < 0.001), Rings task (A) (p < 0.001), Interference (IN) (p < 0.001). This may be due to the transition from the first to the second cycle of primary school, as well as the cognitive maturation that takes place from the 3rd to 5th grades of primary school.

To observe the effect of age and the use of digital activities on the difference in scores obtained between pretest and posttest, the objective of the following section is set out.

Analysis of the Effect of Using Digital Material to Executive Functions

The second goal of the study is to analyze the effects of the use of these activities through the obtained scores via the ENFEN test. First, an ANOVA of repeated measures was applied to explore the differences between the scores obtained in the pretest and posttest. Dependent variables were the scores obtained via ENFEN with pretest and posttest. The independent variable between subjects is students’ grade. Time of usage by different worlds was included as covariable. The results obtained from the analysis are significant in the test of Gray Trails (S1) from the ENFEN battery with the use of educational applications within the following worlds: Logic (N = 145), F(1) = 7.704, p = 0.006 (p < 0.05), p = 0.787, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.052; Spatial (N = 133), F(1) = 7.648, p = 0.007 (p < 0.05), p = 0.784, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.056; and Arts F(1) = 5.092, p = 0.029 (p < 0.05), p = 0.597, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.108 (Figure 4). These results could indicate an effect on tasks working in tandem with executive functions such as spatial perception, processing speed, logical reasoning, working memory, visual-motor efficiency, capacity for anticipation, and foresight through the use of the activities within the worlds of Logic, Spatial, and Arts. In this manner, the hypothesis H0 would be rejected and H1 would be accepted, because there is an effect between the use of digital material from the worlds of Logic, Spatial, and Arts with the executive functions evaluated in the Gray Trails task (S1). No effect on time of use was found for Science activities with Gray Trails task (S1); thus, in this case H0 would be accepted. No effect on time of use was observed for the scores in the rest of the tasks (F1, F2, S2, A, IN). In this case, hypothesis H0 would be accepted, because no effect is found from the use of digital material analyzed with the executive functions that are evaluated in tasks F1, F2, S2, A, and IN.

FIGURE 4

Figure 4 | (A) Graphic of S1 Scores from students that played the Logic World. (B) Graphic of S1 Scores from students that played the Spatial World. (C) Graphic of S1 Scores from students that played the Arts World.

In relation to the results about the differences that may be found among grades, significance is only obtained with Arts: F(2) = 4,392, p = 0.019 (p < 0.05), p = 0.727, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.173. This might be due to differences in time of usage, mode of use by teachers, and the number of students using the applications of this world.

Secondly, in order to deepen the analysis of the effects and the type of relationship that can be found among the variables, a linear regression was carried out on the significant tests analyzed above. In Logic (N = 145), R² = 0.066, Adjusted R² = 0.053, t = 2.958, p = 0.004 (<0.05), β = 0.252. In Spatial (N = 133), R² = 0.071, Adjusted R² = 0.057, t = 2.911, p = 0.004 (p < 0.05), β = 0.246. No significance is obtained in relation to the students’ grade. With the Arts world, no significance is obtained from linear regression, neither with the time of use nor with the grade. This may indicate effects upon Gray Trails (S1) test scores, with increased use of the Logic and Spatial worlds’ games (Figure 5). However, the effects obtained through linear regression are limited, showing a positive trend. Nonetheless, as no significance was found with the use of applications within the Art world, no relationship has been established with the use of these games and the scores obtained from the Gray Trails test (S1).

FIGURE 5

Figure 5 | (A) Graphic of regression analysis with the usage of Logic world and its relation with scores from S1. (B) Graphic of regression analysis with the usage of Spatial world and its relation with scores from S1.

Post hoc Design to Contrast the Effect of the Use of Digital Material on Executive Functions

In order to investigate the effects of the relationship between differences in use and the scores obtained, a post hoc experimental design was carried out, dichotomizing the independent variable of usage by its mean. In this manner, a group with less use recorded (0) and another with more use recorded (1) were created. Thus, the group with less use can serve as a control group and the group with more use can operate as a pilot group. These groups were made for each world (i.e., Science, Logic, Spatial, and Arts). Statistics describing these groups are shown in Table 2.

TABLE 2

Table 2 | Descriptive statistics of children by groups created for the post hoc experiment.

With the aim of analyzing the effect of usage on the dependent variable of the scores obtained from the ENFEN tasks between the two groups, a general linear multivariate model was applied. The statistical results of the MANOVA for the independent variables of dichotomized “time of usage” are Logic F(6) = 1.383, p = 0.226 (p > 0.05); Spatial F(6) = 1.531, p = 0.173 (p > 0.05); Arts F(6) = 0.365; p = 0.896 (p > 0.05); Science F(6) = 0.658; p = 0.683 (p > 0.05). Each one of these groups created by the use of the activities of the worlds analyzes the effect between subjects of each one of the ENFEN tasks; in this way, the results can be compared with the previous analysis. To provide more information about the study, it is noted from the results obtained whether there are differences between the subjects in the pretest and posttest in the different tasks. These results are presented by each of the ENFEN tasks, following the scheme of the previous section.

No relationship of usage between the groups in tasks Phonological fluency (F1) and Semantic fluency (F2) is observed. These tasks could have been worked with applications from the Literacy world when assessing students’ phonological and semantic skills. The scarce usage of the Literacy activities did not allow said analysis. Thus, H0 is accepted because there is no relationship between time of the activities done in the worlds analyzed (Science, Logic, Spatial, and Arts) and scores taken by Phonological fluency and Semantic fluency.

As for the Gray Trails task (S1), a significant difference is observed between the group with the lowest and the highest use of Logic activities, F(2) = 4,605, p = 0.012 (p < 0.05), p = 0.772, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.061, and Spatial, F(2) = 5,378, p = 0.006 (p < 0.05), p = 0.835, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.076 (Figure 6). This is consistent with the results obtained in the analyses of ANOVA of repeated measures. Explorations also found that the effects among subjects were significant in the posttest scores for both types of activities. Similar to analysis before, no relationship was found between the differences in the use of Science activities nor among subjects in the pretest or the posttest. No significant effect was obtained with the differences in the use of Arts activities nor among subjects in the pretest and the posttest. This scenario corresponds to the results of the linear regression analysis. Therefore, H0 is accepted with time of activities done in the worlds of Science and Arts because no relationship is observed on scores taken by the Gray Trails task. In the case of the Logic and Spatial worlds, H0 is rejected and H1 is accepted by obtaining a relationship between more usage and higher scores in Gray Tails task (S1).

FIGURE 6

Figure 6 | Graphics of the differences between group 0 (less use) and group 1 (more use) to scores taken from S1: (A) with the use of Logic activities; (B) with the use of Spatial activities.

In the case of Color Trails task (S2), results only show significant differences among subjects in students who have used the Logic and Spatial applications, both in pretest and posttest results. However, such effects are not obtained with Science or Arts, which may be due to the small number of students from these groups using said applications. In this sense, H0 is accepted, as no relationship was found between the use of digital activities and the scores obtained in the Color Trails task (S2). The same results are obtained in the Interference task (IN), with effect on groups both in the pretest and in the posttest. However, in Interference task (IN) significant differences are also obtained between the groups with greater use and less use with Logic, F(2) = 6,640, p = 0.002 (p < 0.05), p = 0.908, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.086, and Spatial, F(2) = 4,008, p = 0.020 (p < 0.05), p = 0.708, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.058 (Figure 7). Accordingly, H0 is rejected for the use of Logic and Spatial activities. H1 is accepted since the time of use of the activities of the Logic and Space worlds is related to the scores obtained in the Interference task (IN). This relationship with use may be due to the impact of said activities on the executive function of inhibition, but not to that of resistance to interference, which meant that no transference of these was observed in previous analysis. It may be a task not transferred as easy as others to the different types of activities. On the other hand, since effect was found among the subjects in both types of tests, it could indicate differences in the progress of students with inhibition skills or resistance to interference, provided the fact that they are developed at these ages. Consequently, differences among students are likely to be based on their cognitive development.

FIGURE 7

Figure 7 | Graphics of the differences between group 0 (less use) and group 1 (more use) to scores taken at IN (A) with the use of Logic activities (B) with the use of Spatial activities.

In the Rings task, significance is obtained among the groups with different uses of Spatial world activities: F(2) = 3.649, p = 0.029 (p < 0.05), p = 0.664, ${\mathrm{\eta }}_{\text{p}}^2$ = 0.053 (Figure 8). This may indicate a relationship with the processing speed in this test given the reduced time of execution. Thus, H0 is rejected with the use of Spatial. H1 is accepted since the time of use of the Space activities is linked with the scores obtained in the Ring task (A). For the use of the rest of the worlds, H0 is accepted, and no relationship is found between the time of use of the Logic, Science, or Art applications with the scores obtained in the Ring task (A). Effects among subjects show significance among students, given the results obtained in the posttest. However, no effect was obtained from the scores in ANOVA of repeated measures analyses. At this point, it would be advisable to investigate the level of students’ transference and extrapolation of knowledge to different tasks.

FIGURE 8

Figure 8 | Graphics of the differences between group 0 (less use) and group 1 (more use) showing scores obtained from A with the use of Spatial activities. Scores at task A are measured in time, being thus inverse to progress.

Discussion

Discussion of Implementation and Usefulness

The analysis of usage reveals preference for digital material focused on activities that work logic, visual–spatial, science, and artistic skills and knowledge. Nevertheless, no homogeneous usage is found within their lessons. One explanation addresses the lack of time for implementing the platform within their lessons, as reported by teachers during the follow-ups. Some of them were able to introduce digital activities as part of the lessons; meanwhile, others allow students to merely play freely at the end of the lesson’s tasks. This can explain the differences in usage found between Logic and Spatial worlds vs. Science and Arts worlds’ usage. A combination of curricular activities (especially dealing with Logic, Science, and Arts) and non-curricular activities (as Spatial) is then the most used. Similar results were obtained from the analysis of usage at pilot schools from different regions in Spain (Lara Nieto-Márquez et al., 2020). All things considered, preference of usage of materials more suitable as support for the lessons for activities of Logic and Spatial can be observed.

In the analysis of use by grade, differences can be observed between the grade corresponding to the first cycle (3rd of primary) and the grades of the second cycle (4th and 5th). Thus, the use of activities in the second cycle of primary is noteworthy. In the 4th year, there was a more homogeneous use of the activities of the different worlds. In this way, the 4th grade is the year in which the application of the material has been carried out most effectively. This may be due to the amount of content developed for this age group compared to 5th grade, which also had the same teachers. However, in the 3rd grade, yet having a considerable amount of material to be implemented at classroom, time of usage comes lower than for those of the 4th and 5th grades. These differences in use could also be equated by improving teacher training in the use of digital material as support to their class. Likewise, more studies among the educational community would be necessary to determine to what extent this implementation is affected by the teachers’ derivation of use, by the disposition of the material in the classrooms, or by the contents of the digital activities.

Discussion of the Effect of the Use of Digital Material on the Different Tasks of the ENFEN Battery

Results show that an effect with the use of activities of Spatial and Logic may explain the improvement at Gray Trails tasks (S1). No effect of age is found. Moreover, this analysis approaches a positive relation between the higher use of these activities and higher scores at Gray Trails tasks (S1). In post hoc design analysis, this same relationship results. Thus, these kinds of activities can be useful for tasks that involve spatial perception, processing speed, logical reasoning, working memory, visual efficiency, anticipation, and forecasting (Portellano et al., 2011). Activities in the logical–mathematical area include those of calculus and coding that may have contributed to the improvement, as well as those activities based on puzzles from the Spatial world. This coincides with studies, such as that of Castellar et al. (2015), in which students were observed to be able to improve executive functions, relating to working memory, through logical–mathematical skills. A further characteristic that may have influenced task performance comes with playability or the possibility of repeating multiple times activities from this world, thanks to their variability (Lieberman et al., 2009; Diamond, 2012; Blumberg and Fisch, 2013). Other activities, like tales or escape rooms from the Literacy world, are designed to be played once, thus impeding students to train the required skills by repetition.

Furthermore, the use of technology helps in the more effective processing of information, as it is based on visual and auditory channels. This may help to reinforce the executive functions involved in Gray Trails tasks (S1) by storing and handling information in a conscious and active manner (Mayer, 2005; Diamond, 2012, 2013). Likewise, working with visuospatial activities could contribute to a better coordination in this task and competence in spatial processing (Latham et al., 2013). Nevertheless, the training may not have led to develop enough changes at a neuronal level, as for engaging transference of these skills to other tasks within the ENFEN battery (Klingberg, 2010). Improvement accompanying the use of logical–mathematical and spatial applications may be due to a longer time spent, compared to all the other applications in other areas. In this way, literature shows that executive functions, such as reasoning or processing speed, can be improved with digital activities training (Cf. Mackey et al., 2011). Both these areas complement each other, leading to reciprocal improvement, as mathematical and spatial skills share neuronal routes at the prefrontal cortex (Bavelier et al., 2012). Studies such as Wei et al. (2012) or Titz and Karbach (2014) have shown that the training of visuospatial tasks may help to improve such skills, as they correlate in a directly proportional manner to mathematical performance.

In our studies, results suggest that Spatial and Logic activities are useful to improve certain executive functions combining spatial perception, processing speed, logical reasoning, working memory, visual–motor efficiency, anticipation, and foresight (Portellano et al., 2011). In relation to age, only significant differences are observed between the scores obtained from the different tasks of the ENFEN battery. Age is not observed to have an effect along the use of the Logic and Spatial applications. It can be expected to observe a relation with the usage of activities into Logic and Spatial World with other tasks sharing specific executive functions. Obtention of relationships with improvement on Gray Trials (S1) and not with Color Trials (S2) could be explained since in S2, unlike S1, inhibition is involved along with resistance to interference. As observed in the Study 2, there could be some relationship between the Interference task (IN), similar to the Stroop tasks, with the use of Logic and Spatial activities. Nonetheless, no effect of improvement or transference has been detected with ANOVA of repeated-measure analysis (Bull and Scerif, 2001).

Inhibition and resistance to interference are tested on Color Trials (S2) and Interference (IN): their effect with other executive functions might show strong to the point of interfering with the transfer of acquired knowledge. Also, in the Stroop task (in the present study referred as IN) no effect in the use of digital materials can be found on score improvement with ANOVA of repeated-measures analysis. It has also been noticed that the instruction sessions with digital activities improved the performance through the transfer of the training of the working memory, although this same process does not happen to occur with other executive functions, such as inhibition, or resistance to interference. Several studies relate working memory to the capacity for inhibition, as regards the choice of relevant information for the achievement of the task. In this sense, due to joint work, there may be an overlapping effect in our study on these observable functions’ results (Cf. Davidson et al., 2006; Bavelier et al., 2012). Additionally, our not obtaining an effect on inhibition or resistance to interference can be explained by differences among students at these ages, given changes of development at cognitive levels. In post hoc design analysis, there are significances in pretest and posttest in both tasks.

Although there are differences between the scores obtained from the different grades, age effect is not observed in conjunction with the use of digital materials. Thus, significance with the grade is only obtained in Art world activities. On the other hand, the first analysis obtained effect with the variables of grades and usage of digital material. However, when analyzing it to identify the type of relationship, the said effect was lost. In such a way, it could be claimed that the initial effect was related to the students’ worked graphomotricity, which, as the applications of Art do not have a generalized use among students, makes this effect disappear. Likewise, results could indicate a relationship with age and not so much with usage, since in the post hoc design no relationship is obtained either.

Moreover, no effect or relationship is found among the use of the Science world, students’ age, and the ENFEN tasks, perhaps because of their limited use. Likewise, in post hoc design results show relation with the Rings task and use of activities within the Spatial world. The Ring task’s results could indicate an improvement in students’ processing speed, as its score is formulated by the time of performance. Furthermore, a highlight in the improvement of visuospatial skills is the training based on action video games or repetition games (Oei and Patterson, 2013), which are also observed to help improving processing speed. There is no noticeable improvement of any significance with the use of activities from any of the tasks, but there is a reduction in the performance time that could be due to the trained processing speed. One question here arises about whether a significant improvement could be obtained from other executive functions too (inhibition, cognitive flexibility, etc.), that are related to the different ENFEN tests, by increasing the use of all the areas of digital material included in the platform. Improvement in the ENFEN tasks could be due to the enhancement in both processing speed, by reducing the time spent on the tasks, and in accuracy, by reducing the number of mistakes. The balance between processing speed and accuracy varies with age, although no effect was observed in our results (Davidson et al., 2006; Best et al., 2011; Bavelier et al., 2012). A reason for not detecting an improvement in other tasks may be answered exploring the higher effort required for their resolution, lacking transference of other trained skills into this one (Cf. Davidson et al., 2006; Bavelier et al., 2012; Karbach and Unger, 2014).

Limitations of the Study

It is necessary to consider the limitations of the study, for the understanding of the analysis of digital material and its effect, in order to avoid misinterpretations. The importance of executive functions lies in their prediction of children’s academic performance and that these can improve or strengthen their proper development through their reinforcement by instruction in children aged up to early adulthood (Davidson et al., 2006; Karbach and Kray, 2009; Mackey et al., 2011; Diamond, 2012, 2013). There is still a need to continue studying how transference takes place to different daily or school activities, and whether this can be adapted according to students’ individual needs (Karbach and Kray, 2009). There are still limitations in the studies on training and transference depending on the executive functions, tasks being trained, and the goals to be achieved (Bavelier et al., 2012; Karbach and Unger, 2014; Titz and Karbach, 2014). In this case, some transference of the skills worked with the digital material has been observed in tasks such as Gray Trials (S1). This may lead us to question what school tasks could this skills transference be observed with. This study also raises limitations on training and transfer of specific executive functions. That is, task scores may be due to the coordination of multiple executive functions. Another consideration could be that interventions based on working executive functions require more time, as discussed in their study (Sulik et al., 2020). The present assessment hinders an individual rating of the functions trained the most through the different skills, which could have been developed according to the schema of the development of the digital educational material. On the other hand, the effects observed in the linear regression of Study 1 are limited. This effect shows a light trend that digital material could add something to the performance of some tasks. It should also be noted that the variables used in the study are not marked by major changes. Improvement in all the scores may be due to the skills acquired during the year; however, lacking a control group, this observation resists a precise answer and any further explanation (Cf. Mackey et al., 2011). Therefore, a post hoc study was designed to see the comparison assumption between a pilot group and a control group generated. Otherwise, these studies focus actual conditions of usage at school with all daily life troubles, the reason why designing a post hoc study applies for analyzing differences between time of usage.

This study is based on an external evaluation of digital material using a neuropsychological test to assess progress of executive functions. To be able to give a better response to the progress of the students, metrics could be implemented in the platform through the “Learning analytics system.” That point would provide a more exhaustive evaluation of the same variables as the neuropsychological tests such as ENFEN to improve the comparative studies of the quality of the material. In addition, using different programs could improve these evaluations. It would also be interesting to evaluate different pedagogical aspects, teaching, teacher profiles, times, and mode of use, among others. This would help to give more accurate feedback on aspects to be improved. Further research including longitudinal studies is required to verify the effects of long-term training and knowledge transference versus sporadic training or uses. Additionally, extending the study samples would allow an analysis of the diverse variables that may be influencing the learning process, the contexts, and other factors. A study on the transference of these learning processes will help to design interventions to develop the competences required for future management, as yet undefined; e.g., the competence of learning to learn (Cf. Lieberman et al., 2009; Patino et al., 2016).

Sundry studies have sought to analyze executive functions individually (Miyake et al., 2000; Davidson et al., 2006; Lopez-Rosenfeld et al., 2013), which is appropriate for a study that explores and addresses their functioning. Nevertheless, isolating the different executive functions in activities designed for classroom use is a complex matter (Best et al., 2011). Multi-domain tasks, such as the ENFEN battery, assess several executive functions simultaneously, which complicates any specific study of their separate development and their interactions between in skills transference (Karbach and Unger, 2014). The usage of an instrument such as ENFEN can also be a limitation when evaluating multiple cognitive processes within executive functions. The evaluation and application of these tasks are applied according to the publisher’s manual (Portellano et al., 2011; see additional material) and contrasts with theories such as that of Miyake et al. (2000). Although certain pieces of literature (Cf. Miyake et al., 2000) consider the model of study the executive functions as separable components for specificity studies, school tasks however require of a more accurate description of plurality via multi-domain tasks engaging the combined application of multiple executive functions as ensemble (Davidson et al., 2006). The Smile and Learn Platform has designed activities in game, video, and quiz format, similar to the activities that can be posed in the classroom to work on the school curriculum. This manner, the applied ENFEN instrument was selected as being the most appropriate one for the general analysis addressing this educational platform. However, studies of the effect of digital material with elementary school students and materials in Spanish were so scarce. Another limitation that can be pointed out in relation to academic performance is that the intelligence quotient as a variable has not been controlled. However, the heterogeneity and size of the sample, as well as the other variables analyzed, can provide sufficient value and ecological validity to the study.

Conclusion

As commented in the discussion, it is important to consider the limitations of this study for its interpretation, such as the lack of a control group. For the analyses of the data, the times of use correspond to the groups of students who have used the digital activities according to the worlds of Logic, Space, Science, and Art. Thus, for each of these groups of students who have used the digital material, their results have been analyzed in the ENFEN tasks (F1, F2, S1, S2, IN, A).

One of the key aspects comes with our methodology, given it centers the importance of a proper implementation and use in the classrooms to show more precise results or gather more data, as well as a suitable teachers’ training, along with the guides for selecting digital materials that best fit their educational purposes. The aim of many of these materials tends to facilitate teaching practice through resources customization. Teachers may thus exercise and focus on mentoring and enhancing social relationships in the classroom to enrich the learning process (Lopez-Rosenfeld et al., 2013). The classrooms’ usage of Smile and Learn has been freely undertaken by the teachers in order to study the actual implementation this material may have at a school, considering preference and interests upon the different study areas. Establishing a strict use within the school timetable performance could have been improved; nevertheless, by ceasing to be a complementary aspect thus becoming mandatory, it could lose its motivational trait, impacting the students’ performance.

There is still a need to continue investigating for improving the material’s design and assessment in order to achieve specific learning goals, good interactivity, immersion, etc. (Revelle, 2013; Arnab et al., 2015; Abdelali et al., 2016). This stream of research could focus on exploring the relationship and interaction between the gaming elements for learning, in the same way that there are established patterns based on the mechanics of games for prompting actions in players. Furthermore, there is a need to consider the effort and time required when designing and developing digital materials, involving a proper gathering of data through Learning Analytics Systems through intrinsic assessment of games, among other matters (Hickmott et al., 2016). This requires interdisciplinary teams of researchers, educators, designers, and developers of educational applications that should work together to continue improving materials on the market, proposing solutions to limitations in terms of development which involve common vocabulary and more flexible approaches extending beyond each area’s disciplinary perspectives (Cf. Bellotti et al., 2011; Arnab et al., 2012, 2015; Hirsh-Pasek et al., 2015; Hickmott et al., 2016; Arnab and Clarke, 2017).

Systematically, following the design guidelines, the mechanics or elements of the activities of the Logic and Spatial worlds include repetition possibilities, error and success feedbacks, time spent on the task, and rewards. These characteristics, as well as the logical–mathematical and visuospatial skills worked through these activities, may have helped students to improve in tasks involving executive functions such as spatial perception, processing speed, logical reasoning, working memory, visual–motor efficiency, anticipation, and foresight. Based on our results, the design of more variegated games focusing on the training of executive functions and visual–spatial fields is advised. This would help researchers in clarifying the relationships among executive functions and visual–spatial skills, especially those of inhibition and resistance to interference, in different tasks. Deeper analyses on said characteristics will show new ideas for designing further activities for other worlds of the platform.

Further research is needed to define the specific game elements included in these types of activities for determining their link to learning. Despite that studies as Baptista et al. (2015) address performance on skills focusing on operations, decision making, problem solving, or organization with games and quizzes, more research is needed to establish definitive relationships with learning variables. Likewise, it would be advisable to implement built-in metrics that could evaluate executive functions internally, from the design of such activities, as well as neuropsychological tests. Studies such as Homer et al. (2018) have used commercialized games to see the effect of their use on students’ cognitive skills, such as executive functions. This study pointed out the usefulness of some commercialized games as opposed to others developed for analysis in educational interventions. Moreover, the use of digital games and technology may help to build a knowledge base for a subsequent transference of learning, supporting the acquisition of more complex knowledge; nevertheless, this should not be considered the only means (Cf. Blumberg and Fisch, 2013). By including metrics in activities, internal assessment of activities could be contrasted with external assessments to address relationships between different variables. This would allow checking the consequential transference of skills and knowledge among different tasks.

Implications of the Study

As for this study’s approach, our work concerns upon a complete variety of school areas (all curricular areas were included), in contrast with other studies that focused on an isolated specific area. We could find a sheer lack of said open-scope articles in the literature. Thus, this study provides a general external evaluation of digital educational material in the school context. For this purpose, the similarities of the instrument used with the tasks carried out in the different worlds of activities of the platform analyzed were considered. This general study gives an overview of the areas in which the use of technology in schools seems to have a greater interest and effect.

As a result of this interdisciplinary evaluation study on digital material design, UX Research and education could extract other future implications for the industrial and academic sectors:

• Discussing the relationship between our results and said areas, those including logical and visuospatial game activities contributed with the most significant results. Thus, deepening in such games, research would provide more efficient contrast for further comparisons within the literature and design.

• More open-scope studies will be necessary for understanding how these materials get implemented at schools, by trained teachers, and upon specific children niches. In this sense, more research on the differences among areas is required. Digital educational material is far from being adapting and fully incorporating all pedagogical theories in order to obtain significant results at children’s development without any teachers’ support (self-regulated learning).

• Specific results in each of the areas could be completed from the field of cognitive psychology, including more variables of academic performance as intelligence quotient. To this end, it could be recommended to the industrial community to consider both the pedagogical objectives to be included in game design and skills through executive functions. In this way, training studies of specific components of executive functions could be considered through these materials.

• Further research in these areas could help to relate more comprehensively the relationship between the design of digital material and its mechanics to learning processes. This would support the possibility of creating design and development guides useful for EdTech companies.

• Investigate the relationship of executive functions with self-regulated learning supported by digital material.

Data Availability Statement

Datasets are available on request to the corresponding author. The following licenses apply: the privacy policy complies with the Article 13 of the EU 2016/679 Law, April 27th, and the European General Data Protection Regulation (GDPR).

Ethics Statement

This study was carried out following the guidelines in accordance with the Organic Law 3/2018, passed on December 5th, for the Protection of Personal Data and guarantee of digital rights as the privacy policy at Smile and Learn. This project has been approved by the Research Ethics Committee of the University Camilo José Cela. Written informed consent to participate in this study was provided by the participants’ legal guardian/next of kin.

Author Contributions

NL and MP: methodology, investigation, and formal analysis. NL: data curation. NL and AG: resources. NL, AG, and AB: writing—original draft preparation. NL, MP, and AC: writing—review and editing. NL, AC, MP, and AB: visualization. MP: project administration. NL: funding acquisition. All authors read and agreed to the published version of the manuscript.

Funding

This study is part of the project IND2017/SOC-7874 for assessment and improve the digital material design, funded by the Community of Madrid by ‘Industrial Ph.D’ grants.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank the students and teachers from the pilot school who have collaborated with this study. Special thanks are due to Sandra García for her assistance with the respondents.

Supplementary Material

More information about the materials used with the test ENFEN (Portellano et al., 2011) can be found online: http://web.teaediciones.com/ENFEN–EVALUACIONNEUROPSI COLOGICA-DE-LAS-FUNCIONES-EJECUTIVAS-EN-NINOS.aspx

Didactic guides and contents used in this study can be found online: https://smileandlearn.com/schools/?lang=en

References

Abdelali, S., Mateu, S., Imma, B., Fatiha, E., and Mohammed, B. (2016). Improving serious game design through a descriptive classification: a comparation of methodologies. J. Theor. Appl. Inform. Technol. 92, 130–143.

Google Scholar

AENOR (2017). Norma UNE 71362:2017, De 14 De Junio De 2017. Calidad de Los Materiales Educativos Digitales [Digital Didactic Material Quality]. Madrid: Asociación Española de Normalización y Certificación (AENOR).

Google Scholar

Anderson, V. (2001). Assessing executive functions in children: biological, psychological, and developmental considerations. Pediatr. Rehabil. 4, 119–136. doi: 10.1080/713755568

CrossRef Full Text | Google Scholar

Arnab, S., Berta, R., Earp, J., De Freitas, S., Popescu, M., Romero, M., et al. (2012). Framing the adoption of serious games in formal education. Electron. J. E-Learning 10, 159–171.

Google Scholar

Arnab, S., and Clarke, S. (2017). Towards a trans-disciplinary methodology for a game-based intervention development process. Br. J. Educ. Technol. 48, 279–312. doi: 10.1111/bjet.12377

CrossRef Full Text | Google Scholar

Arnab, S., Lim, T., Carvalho, M. B., Bellotti, F., De Freitas, S., Louchart, S., et al. (2015). Mapping learning and game mechanics for serious games analysis. Br. J. Educ. Technol. 46, 391–411. doi: 10.1111/bjet.12113

CrossRef Full Text | Google Scholar

Baptista, R., Coelho, A., and de Carvalho, C. V. (2015). “Relationship between game categories and skills development: contributions for serious game design,” in Proceedings of the European Conference on Game Based Learning, Brighton, 34–42.

Google Scholar

Bavelier, D., Green, C. S., Pouget, A., and Schrater, P. (2012). Brain plasticity through the life span: learning to learn and action video games. Annu. Rev. Neurosci. 35, 391–416. doi: 10.1146/annurev-neuro-060909-152832

PubMed Abstract | CrossRef Full Text | Google Scholar

Bellotti, F., Ott, M., Arnab, S., Berta, R., de Freitas, S., Kiili, K., et al. (2011). “Designing serious games for education: from pedagogical principles to game mechanisms,” in Proceedings of the 5th European Conference on Games Based Learning, (Greece: University of Athens), 26–34.

Google Scholar

Best, J. R., Miller, P. H., and Jones, L. L. (2009). Executive functions after age 5: changes and correlates. Dev. Rev. 29, 180–200. doi: 10.1016/j.dr.2009.05.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Best, J. R., Miller, P. H., and Naglieri, J. A. (2011). Relations between executive function and academic achievement from ages 5 to 17 in a large, representative national sample. Learn. Individ. Diff. 21, 327–336. doi: 10.1016/j.lindif.2011.01.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Blumberg, F. C., and Fisch, S. M. (2013). “Introduction: digital games as a context for cognitive development, learning, and developmental research,” in Digital Games: A Context for Cognitive Development. New Directions for Child and Adolescent Development, eds F. C. Blumberg and S. M. Fisch (Hoboken, NJ: John Wiley & Sons), 1–9.

Google Scholar

Bowie, C. R., and Harvey, P. D. (2006). Administration and interpretation of the trail making test. Nat. Protoc. 1, 2277–2281. doi: 10.1038/nprot.2006.390

PubMed Abstract | CrossRef Full Text | Google Scholar

Brocki, K. C., and Bohlin, G. (2004). Executive functions in children aged 6 to 13: a dimensional and developmental study. Dev. Neuropsychol. 26, 571–593. doi: 10.1207/s15326942dn2602_3

CrossRef Full Text | Google Scholar

Bull, R., Espy, K. A., and Senn, T. E. (2004). A comparison of performance on the Towers of London and Hanoi in young children. J. Child Psychol. Psychiatry 45, 743–754. doi: 10.1111/j.1469-7610.2004.00268.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Bull, R., and Scerif, G. (2001). Executive functioning as a predictor of children’s mathematics ability: inhibition, switching, and working memory. Dev. Neuropsychol. 19, 273–293. doi: 10.1207/S15326942DN1903_3

CrossRef Full Text | Google Scholar

Castellar, E. N., All, A., De Marez, L., and Van Looy, J. (2015). Cognitive abilities, digital games and arithmetic performance enhancement: a study comparing the effects of a math game and paper exercises. Comput. Educ. 85, 123–133. doi: 10.1016/j.compedu.2014.12.021

CrossRef Full Text | Google Scholar

Cheung, A. C., and Slavin, R. E. (2012). How features of educational technology applications affect student reading outcomes: a meta-analysis. Educ. Res. Rev. 7, 198–215. doi: 10.1016/j.edurev.2012.05.002

CrossRef Full Text | Google Scholar

Davidson, M. C., Amso, D., Anderson, L. C., and Diamond, A. (2006). Development of cognitive control and executive functions from 4 to 13 years: evidence from manipulations of memory, inhibition, and task switching. Neuropsychologia 44, 2037–2078. doi: 10.1016/j.neuropsychologia.2006.02.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Diamond, A. (2012). Activities and programs that improve children’s executive functions. Curr. Direct. Psychol. Sci. 21, 335–341. doi: 10.1177/0963721412453722

PubMed Abstract | CrossRef Full Text | Google Scholar

Diamond, A. (2013). Executive functions. Annu. Rev. Psychol. 64, 135–168. doi: 10.1146/annurev-psych-113011-143750

PubMed Abstract | CrossRef Full Text | Google Scholar

García, F. T., González-Castro, P., Areces, D., Cueli, M., and Rodríguez, P. C. (2014). Executive functions in children and adolescents: the types of assessment measures used and implications for their validity in clinical and educational contexts. Papeles Psicól. 35, 215–223.

Google Scholar

Gardner, H. (1983). Multiple Intelligences. New York, NY: Basic Books.

Google Scholar

Gathercole, S. E., Pickering, S. J., Knight, C., and Stegmann, Z. (2004). Working memory skills and educational attainment: evidence from national curriculum assessments at 7 and 14 years of age. Appl. Cogn. Psychol. 18, 1–16. doi: 10.1002/acp.934

CrossRef Full Text | Google Scholar

Goldin, A. P., Hermida, M. J., Shalom, D. E., Costa, M. E., Lopez-Rosenfeld, M., Segretin, M. S., et al. (2014). Far transfer to language and math of a short software-based gaming intervention. Proc. Natl. Acad. Sci. U.S.A. 111, 6443–6448. doi: 10.1073/pnas.1320217111

PubMed Abstract | CrossRef Full Text | Google Scholar

Hickmott, D., Smith, S. P., Bille, R., Burd, E., Stephens, L., and Southgate, E. (2016). “Building apostrophe power: lessons learnt for serious games development,” in Proceedings of the Australasian Computer Science Week Multiconference, New York, NY.

Google Scholar

Hirsh-Pasek, K., Zosh, J. M., Golinkoff, R. M., Gray, J. H., Robb, M. B., and Kaufman, J. (2015). Putting education in “educational” apps: lessons from the science of learning. Psychol. Sci. Public Interest 16, 3–34. doi: 10.1177/1529100615569721

PubMed Abstract | CrossRef Full Text | Google Scholar

Homack, S., and Riccio, C. A. (2004). A meta-analysis of the sensitivity and specificity of the Stroop Color and Word Test with children. Arch. Clin. Neuropsychol. 19, 725–743. doi: 10.1016/j.acn.2003.09.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Homer, B. D., Plass, J. L., Raffaele, C., Ober, T. M., and Ali, A. (2018). Improving high school students’ executive functions through digital game play. Comput. Educ. 117, 50–58. doi: 10.1016/j.compedu.2017.09.011

CrossRef Full Text | Google Scholar

Howieson, D. (2019). Current limitations of neuropsychological tests and assessment procedures. Clin. Neuropsychol. 33, 200–208. doi: 10.1080/13854046.2018.1552762

PubMed Abstract | CrossRef Full Text | Google Scholar

Hunicke, R., LeBlanc, M., and Zubek, R. (2004). “MDA: a formal approach to game design and game research,” in Proceedings of the AAAI Workshop on Challenges in Game AI, Pittsburgh.

Google Scholar

Jaeggi, S. M., Buschkuehl, M., Jonides, J., and Shah, P. (2011). Short-and long-term benefits of cognitive training. Proc. Natl. Acad. Sci. U.S.A. 108, 10081–10086. doi: 10.1073/pnas.1103228108

PubMed Abstract | CrossRef Full Text | Google Scholar

Karbach, J., and Kray, J. (2009). How useful is executive control training? Age differences in near and far transfer of task-switching training. Dev. Sci. 12, 978–990. doi: 10.1111/j.1467-7687.2009.00846.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Karbach, J., and Unger, K. (2014). Executive control training from middle childhood to adolescence. Front. Psychol. 5:390. doi: 10.3389/fpsyg.2014.00390

PubMed Abstract | CrossRef Full Text | Google Scholar

Klingberg, T. (2010). Training and plasticity of working memory. Trends Cogn. Sci. 14, 317–324. doi: 10.1016/j.tics.2010.05.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Lara Nieto-Márquez, N., Baldominos, A., Cardeña Martínez, A., and Pérez Nieto, M. Á (2020). An exploratory analysis of the implementation and use of an intelligent platform for learning in primary education. Appl. Sci. 10:983. doi: 10.3390/app10030983

CrossRef Full Text | Google Scholar

Latham, A. J., Patston, L. L., and Tippett, L. J. (2013). The virtual brain: 30 years of video-game play and cognitive abilities. Front. Psychol. 4:629. doi: 10.3389/fpsyg.2013.00629

PubMed Abstract | CrossRef Full Text | Google Scholar

Lehto, J. E., Juujärvi, P., Kooistra, L., and Pulkkinen, L. (2003). Dimensions of executive functioning: evidence from children. Br. J. Dev. Psychol. 21, 59–80. doi: 10.1348/026151003321164627

PubMed Abstract | CrossRef Full Text | Google Scholar

Lieberman, D. A., Bates, C. H., and So, J. (2009). Young children’s learning with digital media. Comput. Sch. 26, 271–283. doi: 10.1080/07380560903360194

CrossRef Full Text | Google Scholar

Lim, T., Carvalho, M. B., Bellotti, F., Arnab, S., De Freitas, S., Louchart, S., et al. (2015). The LM-GM Framework for Serious Games Analysis. Pittsburgh: University of Pittsburgh.

Google Scholar

Lopez-Rosenfeld, M., Goldin, A. P., Lipina, S., Sigman, M., and Slezak, D. F. (2013). Mate Marote: a flexible automated framework for large-scale educational interventions. Comput. Educ. 68, 307–313. doi: 10.1016/j.compedu.2013.05.018

CrossRef Full Text | Google Scholar

Mackey, A. P., Hill, S. S., Stone, S. I., and Bunge, S. A. (2011). Differential effects of reasoning and speed training in children. Dev. Sci. 14, 582–590. doi: 10.1111/j.1467-7687.2010.01005.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Martins, D. N., and Gotuzo, S. A. (2017). Intervention for executive functions development in early elementary school children: effects on learning and behaviour, and follow-up maintenance. Educ. Psychol. 37, 468–486. doi: 10.1080/01443410.2016.1214686

CrossRef Full Text | Google Scholar

Mateos, M. J., Muñoz-Merino, P. J., Kloos, C. D., Hernández-Leo, D., and Redondo-Martínez, D. (2016). “Design and evaluation of a computer-based game for education,” in Poceedings of the IEEE Frontiers in Education Conference, (Piscataway, NJ: IEEE), 1–8.

Google Scholar

Mayer, R. E. (2005). Cognitive theory of multimedia learning. Camb. Handb. Multimedia Learn. 41, 31–48.

Google Scholar

Meixner, J. M., Warner, G. J., Lensing, N., Schiefele, U., and Elsner, B. (2019). The relation between executive functions and reading comprehension in primary-school students: a cross-lagged-panel analysis. Early Childhood Res. Q. 46, 62–74. doi: 10.1016/j.ecresq.2018.04.010

CrossRef Full Text | Google Scholar

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., and y Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: a latent variable analysis. Cogn. Psychol. 41, 49–100. doi: 10.1006/cogp.1999.0734

PubMed Abstract | CrossRef Full Text | Google Scholar

Oei, A. C., and Patterson, M. D. (2013). Enhancing cognition with video games: a multiple game training study. PLoS One 8:e0058546. doi: 10.1371/journal.pone.0058546

PubMed Abstract | CrossRef Full Text | Google Scholar

Patino, A., Romero, M., and Proulx, J. N. (2016). “Analysis of game and learning mechanics according to the learning theories,” in Proceedings of the 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), Barcelona, 1–4.

Google Scholar

Pérez, M. D. M., Duque, A. G., and Garcia, L. F. (2018). Game-based learning: increasing the logical-mathematical, naturalistic, and linguistic learning levels of primary school students. J. New Approaches Educ. Res. 7, 31–39.

Google Scholar

Portellano, J. A., Martínez-Arias, R., and Zumárraga, L. (2011). ENFEN. Evaluación Neuropsicológica de las funciones ejecutivas en niños [Neuropsychological Assessment of Executive Functions in Children]. Madrid: TEA Ediciones, S.A.U.

Google Scholar

Porteus, S. D. (1933). The Maze Test and Mental Differences. Anoka, MN: The Smith Printing and Publishing House.

Google Scholar

Redick, T. S., Calvo, A., Gay, C. E., and Engle, R. W. (2011). Working memory capacity and go/no-go task performance: selective effects of updating, maintenance, and inhibition. J. Exp. Psychol. 37, 308–324. doi: 10.1037/a0022216

PubMed Abstract | CrossRef Full Text | Google Scholar

Revelle, G. (2013). Applying developmental theory and research to the creation of educational games. New Direct. Child Adolesc. Dev. 139, 31–40. doi: 10.1002/cad.20029

PubMed Abstract | CrossRef Full Text | Google Scholar

Rinat, K., Ora, K., and Andrea, B. (2005). Analysis of word clustering in verbal fluency of school-aged children. Arch. Clin. Neuropsychol. 20, 1087–1104. doi: 10.1016/j.acn.2005.06.012

PubMed Abstract | CrossRef Full Text | Google Scholar

Rutherford, T., Buschkuehl, M., Jaeggi, S. M., and Farkas, G. (2018). Links between achievement, executive functions, and self-regulated learning. Appl. Cogn. Psychol. 32, 763–774. doi: 10.1002/acp.3462

CrossRef Full Text | Google Scholar

Sajjadi, P., Vlieghe, J., and De Troyer, O. (2016). “Evidence-based mapping between the theory of multiple intelligences and game mechanics for the purpose of player-centered serious game design,” in Proceedings of the 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), Barcelona, 1–8.

Google Scholar

Santiago, K., Lukas, J. F., Etxeberria, J., and Gobantes, A. (2009). Evaluation of the IKASYS programme. Research, Reflections and Innovations in Integrating ICT in Education. Badajoz: FORMATEX, 51–54.

Google Scholar

Siang, A. C., and Rao, R. K. (2003). “Theories of learning: a computer game perspective,” in Proceedings of the Fifth International Symposium on Multimedia Software Engineering, Taichung.

Google Scholar

Simmonds, D. J., Pekar, J. J., and Mostofsky, S. H. (2008). Meta-analysis of Go/No-go tasks demonstrating that fMRI activation associated with response inhibition is task-dependent. Neuropsychologia 46, 224–232. doi: 10.1016/j.neuropsychologia.2007.07.015

PubMed Abstract | CrossRef Full Text | Google Scholar

Slussareff, M., Braad, E., Wilkinson, P., and Strååt, B. (2016). “Games for Learning,” in Entertainment Computing and Serious Games. Lecture Notes in Computer Science 9970, eds R. Dörner, S. Göbel, M. Kickmeier-Rust, M. Masuch, and K. Zweig (Cham: Springer).

Google Scholar

Sulik, M. J., Finch, J. E., and Obradović, J. (2020). Moving beyond executive functions: challenge preference as a predictor of academic achievement in elementary school. J. Exp. Child Psychol. 198:104883. doi: 10.1016/j.jecp.2020.104883

PubMed Abstract | CrossRef Full Text | Google Scholar

Tan, P. H., Ling, S. W., and Ting, C. Y. (2007). “Adaptive digital game-based learning framework,” in Proceedings of the 2nd International Conference on Digital Interactive Media in Entertainment and Arts, New York, NY, 142–146.

Google Scholar

Titz, C., and Karbach, J. (2014). Working memory and executive functions: effects of training on academic achievement. Psychol. Res. 78, 852–868. doi: 10.1007/s00426-013-0537-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Wei, W., Yuan, H., Chen, C., and Zhou, X. (2012). Cognitive correlates of performance in advanced mathematics. Br. J. Educ. Psychol. 82, 157–181. doi: 10.1111/j.2044-8279.2011.02049.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhang, D., Zhou, L., Briggs, R. O., and Nunamaker, J. F. Jr. (2006). Instructional video in e-learning: assessing the impact of interactive video on learning effectiveness. Inform. Manag. 43, 15–27. doi: 10.1016/j.im.2005.01.004

CrossRef Full Text | Google Scholar

Zysset, S., Müller, K., Lohmann, G., and von Cramon, D. Y. (2001). Color-word matching Stroop task: separating interference and response conflict. Neuroimage 13, 29–36. doi: 10.1006/nimg.2000.0665

PubMed Abstract | CrossRef Full Text | Google Scholar