Fostering Health Education With a Serious Game in Children With Asthma: Pilot Studies for Assessing Learning Efficacy and Automatized Learning Personalization

Delmas, Alexandra; Clement, Benjamin; Oudeyer, Pierre-Yves; Sauzéon, Helène

doi:10.3389/feduc.2018.00099

ORIGINAL RESEARCH article

Front. Educ., 14 November 2018

Sec. Digital Education

Volume 3 - 2018 | https://doi.org/10.3389/feduc.2018.00099

Fostering Health Education With a Serious Game in Children With Asthma: Pilot Studies for Assessing Learning Efficacy and Automatized Learning Personalization

$\r\nAlexandra Delmas*$ Alexandra Delmas¹^*

Benjamin Clement¹

Pierre-Yves Oudeyer^1,2

Helène Sauzéon^1,3^*

¹Equipe Flowers, Inria, Talence, France
²Ensta Paris-Tech, Paris, France
³Laboratoire Handicap, Activité, Cognition et Santé (EA4136), Université de Bordeaux, Bordeaux, France

Coupled with Health Education programs, an e-learning platform—KidBreath—was participatory designed and assessed in situ (Study 1) and was augmented and tested with an Intelligent Tutoring System (ITS) based on Multi-Armed Bandit Methods (Study 2). For each study, the impact of KidBreath practice was assessed in children with asthma in terms of pedagogical efficacy (knowledge of the illness), pedagogical efficiency (usability, type of motivation and level of interest elicited), and therapeutic effect (illness perception, system's expectation and judgement in disease self-management, child's implication in study). For the Study 1, asthma children aged 8 to 11 years used the tool at home without time pressure for 2 months according to a predefined learning sequence defined by the research team. Results supported pedagogical efficacy of KidBreath, with a significant increase of general knowledge about asthma after use. It also featured a greater learning gain for children knowing the least about the illness before use. Results on pedagogical efficiency revealed a great intrinsic motivation elicited by KidBreath showing a deep level of interest in the edutainment activities. Study 2 explored an augmented version of KidBreath with learning optimization algorithm (called ZPDES) after its use during 1 month. Pedagogical efficacy was less conclusive than Study 1 because less content was displayed due to algorithm parameters. However, the ITS-augmented KidBreath use showed a strong impact in pedagogical efficiency and therapeutic adherence features. Even if implementation improvements must be done in future works, this preliminary study highlighted the viability of our methods to design an ITS as serious game in health education context for all chronic diseases.

Introduction

According to World Health Organization (WHO), asthma is the most common chronic disease among children¹, with a low illness control (Rabe et al., 2004; Gustafsson et al., 2006). One reason for asthma children's multiple hospital stays lies is the lack therapeutic adherence (to have a positive attitude toward good health practices) (Morris and Schulz, 1992). Only 50% of children with asthma are generally known adherent (Bauman et al., 2002). One of the key factors behind a low level of therapeutic adherence usually lies in the patient's lack of knowledge concerning the illness and its medication (Lieberman, 2001; de Benedictis and Bush, 2007; Kharrazi et al., 2008; Naimi et al., 2009).

To improve therapeutic adherence mechanisms, Health education—related to the concept of Health Literacy²–is generating a plethora of research questions to be answered by multidisciplinary teams, including the fields of Education, Health, and Technology-Enhanced Learning (TEL) (Nutbeam, 1998, 2008; Mitic et al., 2012). Therapeutic education aims to help patients suffering from chronic illnesses acquire and maintain self-management skills concerning the awareness and regulation of their illness in their daily life. In order for this apprenticeship to be the most efficient, it has to rely strongly on patients' background and past experiences with the illness (Mitic et al., 2012). A systematic review revealed benefits of educational interventions for self-management for improving health results (Guevara et al., 2003).

Within the TEL community, numerous educational devices and interactive systems were developed to help teachers make children learn better as well as to increase their motivation to learn (Hirsh-Pasek et al., 2015). Massive Open Online Courses (MOOCs)S created using a variety of technological devices and methods, currently spread knowledge worldwide by using a distant, high number of learners to one teacher, asynchronized learning environment. Nowadays, the majority of such online learning systems became much simpler online tools delivering personalized transmedia content to targeted learners, instead of more traditional reading materials students ought to go through. New systems are far from the complex teaching aids initially used in traditional classroom settings. Computer Assisted Education (CAE) systems have evolved as learning tools providing students with encoded set of exercises and associated solutions (Huertas and Juárez-Ramírez, 2013). CAE systems usually provide problems with a single correct answer, hence when the problem arises, a more complex scenario cannot be completely evaluated, especially when a single problem may have a lot of potential correct answers. The effectiveness of CAE systems was shown, but there is still a gap to be filled for more complex topics. This is why the Intelligent Tutoring Systems (ITS) have proved to match and even surpass CAE system performance (Glickman and Dixon, 2002). As learners generally study structurally-similar courses in a linear manner (which accounts for nearly 90% of school dropout behaviors), ITS were created to make learning more accessible and efficient to learners, as well as to define specific pedagogical goals and metrics to be used in education research (Anderson et al., 1995; Koedinger et al., 1997; Nkambou et al., 2010). In the past decades, such systems aimed to automatically evaluate students' learner models. A learner model can be defined by a set of beliefs a teacher can form about a student's level in a specific learning context. Among such beliefs are a set of knowledge and skills to acquire on a target domain and a student's learning preferences.

Considering the massive expansion of the world wide web high-tech devices home usage and the known benefits of young learners' usage of video games in learning contexts, the field of TEL soon welcomed a new sub-field called serious games (Malone, 1980; Cordova and Lepper, 1996; Bergeron, 2006; Kam et al., 2008; Hirsh-Pasek et al., 2015), which were games designed in a traditional video game landscape, but including some pedagogical goals (Kapp, 2012). They are specifically aimed at using intrinsic gaming mechanisms to foster long term learning retention (Whyte et al., 2015). According to the Social Cognitive Theory, intrinsic motivation requires the presence of an intrinsic contentment to involve oneself in a specific activity for one's own benefit (Touré-Tillery and Fishbach, 2014; Kaloti-Hallak et al., 2015).

The main focus was to study the potential benefits of using a serious game designed following a Participatory Design (PD) approach for supporting a therapeutic education intervention amongst children with asthma. Our main contributions are the following:

- The KidBreath assessment in real-life setting (use at home) for its impact in terms of pedagogical efficacy, pedagogical efficiency, and therapeutic effect (Study 1);

- The assessment of the added value of an ITS embedded in KidBreath with a field study according to similar evaluation criteria to those used in study 1 in order to move the field with an automatized learning personalization in health education (Study 2).

Study 1: Field Study for Assessment of KidBreath

Related Work: E-Education for Asthma Children

In order to promote therapeutic adherence, therapeutic educational programs were designed in workshops that could include one or several health practitioners. To foster educational impact in long-term use, researchers (Rikkers-Mutsaerts et al., 2012) carried out internet-based self-management for children with asthma, which included asthma information, news, frequently asked questions and interactive communication with a specialized nurse (Gibson et al., 2002). In another study, the intervention included just internet-based management and education received though the web site with tele-monitoring (Chan et al., 2007). Jan and colleagues studied over a 12 week-period the impact of Internet-based interactive asthma educational and monitoring program, composed of basic information asthma child-related, an electronic diary, an action plan for the patients, and a retrieval analysis system to review the accumulated data on symptoms score (Jan et al., 2007). After using these web-based educational programs from 3 months to 1 year, all studies showed that the quality of life and treatment use were improved but not for asthma control.

According to TEL field, gamification is widely used as an approach due to its known benefits on different aspects of learning (Malone and Lepper, 1987; Cordova and Lepper, 1996; Shaffer, 2006). It refers to the use of game-based mechanisms in a non-gaming context to engage users, to increase their motivation, and to improve learning (Deterding et al., 2011; Kapp, 2012; Hamari et al., 2014). Nowadays, a majority of such tools are designed specifically for the treatment of chronic illness in terms of illness prevention and illness awareness. However, few were evaluated scientifically in terms of knowledge acquisition process and day-to-day management in mid to long term use (Lieberman, 2001; Shegog et al., 2001; Beale et al., 2007; Charles et al., 2007; Kato et al., 2008). These serious games are reported as providing benefits to minimize the pain while hospital staff changed the bandages for burned victims (Sharar et al., 2007; Hoffman et al., 2008), increasing self-esteem, self-management, and foster health practice for diabetes patients (Brown et al., 1997; Lieberman, 2001), and improving self-esteem and compliance for teenage patients suffering from cancer (Beale et al., 2007; Kato et al., 2008).

Regarding impacts for asthma children, only one serious game was studied, i.e., Bronkie the Bronchiasaurus (Lieberman, 1997, 2001). It is a Nintendo video game. The story tale revolves around a prehistoric world covered in dust. One of the main vacuum cleaners had the misfortune to break, therefore letting dust settles all over the ground. Players can perform the following tasks: take their inhaler regularly and in a safe manner, try to avoid asthma attacks provoked by close-contact to dust and fumes, and keep following their quest. In order to progress to the next stage of the quest, children also ought to answer a set of questions about asthma correctly (multiple choice questions on one's respiratory system, asthma self-management, identification of main triggers, instant recognition of first symptoms and how to act in case of emergencies, knowledge of the medicine's active principles, etc.). Lieberman's research team led several studies on the impact of the videogame on therapeutic adherence. One of them asked young asthma patients to play for 40 min at the hospital, complete questionnaires about asthma before and after use, as well as answering a phone interview 1 month later. Results showed an increase in self-esteem and self-efficacy, patients' ability to communicate about illness with their parents, and their general knowledge about asthma and self-regulation (Lieberman, 2001). Despite the great interest of this singular serious game for asthma children, several limitations could be pinpointed. First, all the assessment has been done within a hospital rather than in mainstream environments (such as a child's home) as the best place for health education. Second, there is no mention of how the learning modules of the games were linked and validated with the current therapeutic learning programs involved. For example, none of the proposed activities attempted to capture children's learning curves or the establishment of new daily routines after using the tools. Third, the learning benefit has been assessed for a short-term window, so near/far effects of the serious game have not been documented. Therefore, to overcome these limitations, study 1 aimed to detail design process of a new serious game, i.e., KidBreath, for home's use by asthma children and to assess its pedagogical impact over short-term (1 month) and long term (2 months) period.

KidBreath Design (Details in the Supplementary Material, Appendix 1)

Participatory Design

Our thereby designed e-learning platform, KidBreath proposes asthma-related edutainment activities to asthma patients aged 8 to 10. Complying to user-centered design approaches, a Participatory Design (PD) methodology was selected (Muller and Kuhn, 1993; Sanders, 2002; Melonio and Gennari, 2012). PD aims to build interactive tools by including end-users and relevant stakeholders (for gathering plural experience and expertise) as design partners in various parts of the design process in order to guaranty the conformity of the requirements elicitation process to the user's needs. The team of design partners may be formed of a set of target users (i.e., asthma patients aged 8 to 10) and indirect actors within the learning process (i.e., parents, therapeutic educators, health specialists, or designers). While the use of the PD approach revealed beneficial in several health education studies, none seemed to have been previously applied to the use of serious games for health education in a child population.

The public safety aspects of this project were overseen by the Therapeutic Education Center in Bordeaux Aquitaine (Centre d'Education Thérapeutique de Bordeaux Aquitaine), a health association. Health specialists and young patients both helped in the design of the edutainment activities. Their participation included the creation of a public health education program concerning asthma, iteratively validated by experts and patients during the various steps of the design process. Prior to launching the experimental study, several administrative authorizations were gathered in terms of ethics (COERLE³ within the INRIA), web usage and data gathering (CNIL⁴), National education services (DSDEN⁵) in addition to every child's parental consent form.

To ensure pedagogical benefits of such a tool, KidBreath was evaluated on non-asthma children with the age of our target users – 8 to 9 years old—from a local primary school. Here, learning activities could be chosen (Figure 1) and experiments were conducted in school for 2 weeks to assess the tool impact in terms of pedagogical efficacy (knowledge asthma-related) and efficiency (curiosity, motivation, usability, user experience) (Delmas et al., 2018). Results showed a significant increase of general knowledge about asthma after use, revealing the pedagogical efficacy of KidBreath. Results on pedagogical efficiency revealed intrinsic motivation generated during the study and a good usability of the tool with children showing a deep level of interest in the edutainment activities. Taken together, results were supportive of KidBreath as future device for health education relative to asthma.

FIGURE 1

Figure 1. Visualization of the ≪ My Journey ≫ interface in a choice-condition (Left). By clicking on the first theme in Riddles Tab (My Body), learning activities associated were displayed for each level of difficulty (Right), where the first one is a quiz.

KidBreath Contents

Based on pedagogical techniques used in Patient Therapeutic Education workshops (ANAES, 2002; Nguyen et al., 2003) KidBreath prototype is divided into four main units: “My Journey,” “On Discovering the Galaxy,” “My Diary,” and “My profile” (Figure 1).

My Journey (rocket icon): progressive learning units on the four knowledge sub-domains of asthma (general disease knowledge, biomedicine, symptomatology, and treatments) consists of quizzes and MCQ in a “Riddles” tab, and mini-games in the “Games” tab as shown is Figure 1-left. In order to facilitate the interface understanding and usability a dual mode of entry (text and sound) is proposed where online characters read aloud both questions and answers (Figure 1-right). The unit's sub-activities were defined as such: My Body (Biomedicine), My Warnings (Symptoms), My Tips (General disease Knowledge), and My Medicine (Treatments).

Within a learning activity, the system proposed one determined learning process where (1) the character asked questions concerning asthma; (2) the child answered by free recall-questions to be imputed via a keyboard (for quizzes), via multiple-choice questionnaires or via mini-games, with the possibility to confirm or overturn before final validation; and (3) instantaneous feedback was provided to the child through the game. The character indicated the correct answer (correct or incorrect) along with detailed explanations and facts the children ought to know already (the same feedback was used whether the child answered correctly or not to take into account answers provided randomly).

On Discovering the Galaxy (planet icon): discovery animated videos units on asthma, which composed of answers to the questions frequently asked by the children having participated to KidBreath design and everyday-life situations children may encounter, their consequences on asthma regulation and prevention, and how to avoid critical development of asthma.

My Diary (book icon): proposes to keep track of the child's achievements within KidBreath.

My profile (cosmonaut icon): gathers every bit of information concerning the child-user.

From the KidBreath conception, the following field study explored the impact of its usage in asthma children, where main research contributions are an evaluation in real-life setting (use at home) of an asthma-related serious game designed with a PD and therapeutic education pedagogical methods in terms of pedagogical efficacy (knowledge around the illness), pedagogical efficiency (intrinsic motivation during learning), therapeutic efficacy (illness perception and therapeutic adherence during use).

Methods

Participants

We recruited volunteers' asthma families after explained user study. Some families were recruited in the health association where they carried out therapeutic educational programs during workshops, and others were called after they accepted when their practitioner told them the goal of the project (working in the association). Consequently, 14 asthma families were recruited and including 8 males, 6 females, aged from 7 to 11 years. All the children exhibited a moderate asthma severity (i.e., between the stage 2 and 3 out of 4 of asthma severity, with a use of daily background inhaler) according to physicians of health association.

Inclusion and exclusion criteria have been established. First, children participating in the study had to have been born between 2006 and 2010. These years were included because children younger don't know read, and children older are in middle school. Second, they had not to be home schooled, and shall know basic computer uses. Third, in order to avoid bias due to literacy issues, only French-speaking children not suffering from cognitive impairments and having no literacy or numeracy conceptual issues were eligible. Four, as the study took place in three sessions at the children's home, only children participating to all sessions were considered for the study. For this reason, final analyses therefore accounted for 14 control children.

Regarding ethics, all child-related data, the study's protocol and the entire set of data collected during the experimental study were pre-validated and registered with the following ethics comities:

• INRIA's ethics comity for experimental studies on humans (COERLE n°2016-010)⁶

• CNIL⁷ user studies database

• The head of office of the departmental administration office for education (SDEN)⁸

• The director of the public school where the study took place.

Consent forms were also filled by all children and their parents (or guardians) prior to using KidBreath.

Material

For controlling learning path across participants, learning units were displayed in a predefined sequence. The riddles were firstly activated by theme category (Biomedical, Symptoms, General Knowledge, Treatments), in respect of the exercises type (quiz, MCQ), of the difficulty level (1: easy, 2: medium, 3: hard) and of ID number of activity. After the riddles completion, the Games became active by theme category (Biomedical, Symptoms, General Knowledge, Treatments), in respect of the difficulty level (1: easy, 2: medium, 3: hard) and number of activity. Figure 1 illustrated this sequence as if the user completed activities displayed from top to bottom and from left to right.

When the participant pressed the button “next” after the correction, the next activity displayed. They could switch to other units, but if he wanted return to play in “My journey,” the learning activity in progress was also displayed.

Procedure and Measurement

Because KidBreath is a web application available for computers, tablets and smartphones, each child was told to use KidBreath at least 10 min per week over 2 months on their own device, after giving to them URL address and instructions. The study was divided into 3 assessment sessions over a 2-month period: pre-intervention session (initial assessment), post-intervention session after 1-month use of KidBreath (mid-point assessment) and post-intervention session after 2-month (end-point assessment). Between each month (M_0.5 and M_1.5), participants received by regular mail one questionnaire (i.e., adherence questionnaire) having to-be filled before the next experimenter visit (planned 2 weeks after) (Table 1).

TABLE 1

Table 1. Procedure and timeline of assessment sessions of KidBreath.

Pedagogical efficacy measurement

Before and after using KidBreath, the Asthma Knowledge Questionnaire (AKQ) (Al-Motlaq and Sellick, 2011) was completed by all children. This English-speaking (true or false) questionnaire for asthma patients is generally divided into 24 affirmations about asthma. Once translated into French, a third option was added to all queries (“I don't know”) to remove uncertainty about the child's random choice of an answer (Nguyen et al., 2003).

In order to check whether the children's knowledge could be correlated to their own general knowledge or the sole use of KidBreath informative content, children were asked to fill a Diabetes Knowledge Questionnaire (DKQ)⁹ questionnaire on diabetes. This French-speaking questionnaire was adapted for a better understanding by 8 and 9-year-old children—items 4, 8, 9, 12, and 13 were removed, others were reformulated, the option “I don't know” was included—leaving us with a 10 items single choice questionnaire.

Pedagogical efficiency measurement

The pedagogical efficiency was probed thanks to questionnaires assessing elicited motivation and curiosity, respectively.

–Motivation. To the kind of motivation elicited in children when they use KidBreath. Vallerand's questionnaire (Vallerand et al., 1989, 1992), based on Deci and Ryan's Self-Determination Theory (1985) and commonly used to ascertain the elicitation of intrinsic and extrinsic motivation (e.g., Desrochers et al., 2006) was selected. Once adapted to a public of children users playing a serious game, children had to answer a 21 items questionnaire on their experience during play. Items were visualized by groups of three representing one of the following type of motivation subcomponent:

• Amotivation (AM). The student is amotivated when he doesn't see how the results applied are related to his actions. This shows a lack of motivation.

• Extrinsic Motivation (EM). This kind of motivation is divided into three subcomponents, i.e., External regulation (behavior is regulated through rewards; e.g., passing a grade or a level with honors), Introjection (learner begins to internalize external constraints; e.g., show his smarts to others), and Identification (the chosen ways to achieve external means, such as helping a friend suffering from asthma).

• Intrinsic Motivation (IM) including three subcomponents, i.e., To Know (epistemic need to know and understand, satisfaction of learning for its own sake), Accomplishments (interact with the environment to feel competent, focus on the process of achieving rather than on the outcome), and Feelings (engage in learning to experience stimulating feelings (e.g., sensory pleasure, excitement, fun).

Scores on motivation span from 0 to 3 in each category of items, presented randomly to children as to avoid learning effects.

–Curiosity. Children's level of elicited curiosity were computed by questioning (Gordon et al., 2015): the child was directed to ask as many questions as wishes on a specific topic without the answers being given by the interviewer (Harris, 2012). After using KidBreath children went through this process, asking for additional information on asthma for KidBreath to answer in the future. Scores on curiosity finally range from 0 to the maximum of questions asked.

KidBreath adherence measurement

The KidBreath adherence was probed with a self-made questionnaire. A 4 points Likert scale (score from 0 to 3) was especially self-made for scoring the parents and children's points of views about the system. It was composed of 3 quantitative items: (1) Parents perception of Child engagement at pre-use and post-use of KidBreath: ≪ Do you think your child will be/was cooperative? ≫ (Never/Seldom/Often/More often); (2) Child perception of engagement at pre-use and post-use of KidBreath: ≪ Do you think you will achieve/achieved playing KidBreath weekly for 2 month/since study began? ≫ (Not at all/From time to time/Many times/All the time); (3) Child perception of KidBreath efficacy in pre-use and post-use: ≪ How much do you think KidBreath will help/helped you for better taking your medication, being more careful and having less asthma attacks? ≫ (Not at all/Slightly/Much/Widely).

Therapeutic measurement

It included an assessment of illness perception thanks to Children Illness Perception Questionnaire (CIPQ based on Leventhal's illness representation theory, Leventhal et al., 1984). The CIPQ is a 26-item instrument adapted from the IPQ for adults (Weinman et al., 1996). Like the IPQ, the CIPQ consists of two separate sections: illness beliefs, where the overall score represents the degree to which the illness is perceived as threatening, and children's beliefs regarding the causal aspects of their condition. According to Weinman, Petrie, Sharpe, and Walker, the causality items were not to be regarded as a subscale, so analyses were undertaken only on the first section (Weinman et al., 2000).

Table below summarizes the experimental procedure and timeline of assessment sessions and their content.

Statistical Analyses

All data analyses were performed on IBM SPSS Statistics 20. A significance level of 0.05 was used throughout the analysis process.

Descriptive statistics were computed from raw scores on each questionnaire for each participant. For any mean comparison, the variance equality was checked for performing parametric statistics.

Regarding pedagogical efficacy, results were treated in two steps for catching the short-term effects of KidBreath (initial assessment vs. mid-point assessment) and its long-term effect (initial assessment vs. end-point assessment). For each step, KidBreath's pedagogical efficacy was evaluated using a two-way mixed ANOVA [Time x Disease questionnaire] with pre-post intervention effect on children's AKQ and DKQ scores. Complementary analyses were performed using an independent t-test with pre-post intervention effect for both disease questionnaires. Correlations were performed in order to highlight any possible association between learning gain and scores in pre-use or post-use.

Concerning pedagogical efficiency, KidBreath's effect on motivation was measured according to the results of the Motivation questionnaire [two-way repeated ANOVA with Motivation type (AM vs. EM vs. IM) and Time (M₁ vs. M₂) as independent variables)]. Curiosity scores were also compared between mid-and end-point assessments as another efficiency measure.

Correlations were performed in order to highlight any possible association between pedagogical efficacy and efficiency measures on the following set of variables: measures associated to asthma-related knowledge (learning-gains scores) and measures associated to elicited motivation scores.

Finally, one-way repeated ANOVAs (Time) were applied on CIPQ scores and those from the KidBreath adherence questionnaire. To complete quantitative data, we added most frequent verbatim reported by parents during interviews.

Results

Pedagogical Efficacy

Individual scores across time

Figures 2A, 3A illustrate the distribution of pre- and post- use learning scores on the AKQ and DKQ questionnaires when coupled with the following relative learning gains [(post-use score – pre-use score) / (post-use score + pre-use score)].

FIGURE 2

Figure 2. (A) Child's performance (percentage) for AKQ in M₀ (pre-use), M₁ (mid-point assessment), M₂ (end-point assessment) and relative learning gains. (B) Child's performance (percentage) for “I don't know” answers in AKQ for M₀ (pre-use), M₁ (mid-point assessment), M₂ (end-point assessment).

FIGURE 3

Figure 3. (A) Child's performance (percentage) for DKQ in M₀ (pre-use), M₁ (mid-point assessment), M₂ (end-point assessment) and relative learning gains. (B) “Child's performance (percentage) for I don't know” answers in DKQ for M₀ (pre-use), M₁ (mid-point assessment), M₂ (end-point assessment).

In Figure 2A, learning gains gathered in M₁ session are ranked from lowest to highest. A negative correlation was found between the pre-use AKQ scores and the associated relative learning gains (r = −0.70; p = 0.005). Thus, children who knew the least about asthma before using KidBreath benefited the most in terms of asthma-related knowledge gained.

Complementary analyses were carried out to check if children answered in a random way, or if they knew they did not know asthma knowledge by choosing “I don't know” as an answer on both questionnaires (Figure 2B). A positive correlation was found between the pre-use “I don't know” scores in AKQ and the associated relative learning gains (r = 0.82; p < 0.001). This indicates that children who knew the less about asthma were honest from the beginning and they did not answer randomly.

Concerning DKQ, Figure 3A was ranked according to children results on the AKQ in order to visualize whether the high-ranking children from Figure 2A were also present in the high rankings of Figure 3A.

There was no correlation between the gain of knowledge on asthma and diabetes learning scores. This indicates that learning curves for asthma and for diabetes did not share relationships, as expected.

As also expected, results from DKQ scores were associated to a high random answer behaviors irrespective of assessment session conditions. Complementary analyses about “I don't know” answers in DKQ didn't show difference between the M₀ (52.8%), M₁ (52.1%), and M₂ (46.4%) sessions, which supported the no learning-gain for the diabetes knowledge (Figure 3B).

Short-term effect of KidBreath (M₀ vs. M₁)

A two-way mixed ANOVA [2 (time: M₀ vs. M₁) x 2 (questionnaire: AKQ vs. DKQ)] only revealed questionnaire effect [F_(1,13) = 59.21; p < 0.0001; η² = 0.820] indicating that a majority of children showed better score on asthma (61.5%) than on diabetes (35.7%) questionnaire. Although the time effect as well as Time x Questionnaire effect didn't reach the significance (respectively, p > 0.08; p > 0.100), a time-related increase of AKQ score was observed on AKQ score [t₍₁₃₎ = 3.49; p = 0.004], but not for DKQ score (p > 0.600) (Figure 4A).

FIGURE 4

Figure 4. (A) Means and standard deviation for AKQ and DKQ score as a function of time (pre- vs. M₁ post-use). (B) Means and standard deviation of children's “I don't know” answers for AKQ and DKQ as a function of time (pre- vs. M₁ post-use).

According to “I don't know” answers in AKQ and DKQ (Figure 4B), a two-way mixed ANOVA [2 (time: M₀ vs. M₁) x 2 (questionnaire: AKQ vs. DKQ)] showed questionnaire effect [F_(1,13) = 43.42; p < 0.0001; η² = 0.770] and interaction effect [F_(1,13) = 7.12; p < 0.02; η² = 0.354], but no time effect was found (p > 0.100). Pairwise comparisons only revealed a significant time-related decrease of “I don't know” answers in AKQ (23.9 vs. 14.3%) [t₍₁₃₎ = 2.60; p < 0.03] (Figure 4B).

Overall results from mid-point assessment indicated both children enhanced their asthma knowledge after using KidBreath and they gained confidence in their answers.

Long-term effect of KidBreath (M₀ vs. M₂)

A two-way mixed ANOVA [2 (time: M₀ vs. M₂) x 2 (questionnaire: AKQ vs. DKQ)] showed over a 2-month period a significant questionnaire effect [F_(1,13) = 62.20; p < 0.0001; η² = 0.827], a time effect [F_(1,13) = 9.24; p < 0.01; η² = 0.416] and an interaction effect [F_(1,13) = 4.39; p = 0.05; η² = 0.253] (Figure 5A). Thus, all the children clearly exhibited greater performance over 2-month period for AKQ scores than in DKQ scores.

FIGURE 5

Figure 5. (A) Means and standard deviation of children's pre- and M₂ post-use questionnaires (AKQ and DKQ). (B) Means and standard deviation of children's “I don't know” answers pre- and M₂ post-use questionnaires (AKQ and DKQ).

To check if children answered in random way, a two-way mixed ANOVA [2 (time: M₀ vs. M₂) x 2 (questionnaire: AKQ vs. DKQ)] showed a questionnaire effect [F_(1,13) = 29.40; p < 0.0001; η² = 0.693] and time effect [F_(1,13) = 7.48; p < 0.02; η² = 0.365], but no interaction effect (p > 0.100) (Figure 5B). Pairwise comparisons revealed a significant decrease of “I don't know” answers in AKQ across the time conditions (23.9 vs. 9.6%) [t₍₁₃₎ = 3.82; p = 0.002] but not for DKQ scores (p > 0.200).

Overall results from end-point assessment indicated both children clearly increased their asthma knowledge after using KidBreath and they were more confident in their answers.

Pedagogical Efficiency

For motivation scores, a two-way mixed ANOVA [2 (time: M₁ vs. M₂) x 3 (motivation type: AM vs. EM vs. IM)] showed a significant motivation effect [F_(2,26) = 27.35; p < 0.0001;η² = 0.855] but neither time effect (p > 0.600) nor interaction effect (p > 0.300) were found. Pairwise comparisons revealed this effect was in the favor of the IM condition regardless the time period [M₁: t_{IM vs. AM(13)} = 5.25; p < 0.0001; t_{IM vs. EM(13)} = 5.92; p < 0.0001; t_{EM vs. AM(13)} = 2.98; p = 0.011; M₂: t_IMvs.AM(13) = 4.46; p = 0.001; t_IMvs.EM(13) = 8.72; p < .0001], which revealed an intrinsic motivation was generated during KidBreath's use across the 2-month use (Table 2).

TABLE 2

Table 2. Means (standard deviation) scores on Motivation and Curiosity measures (N = 14).

To complete pedagogical efficiency with the curiosity indicator scores, one-way repeated ANOVA [2 (time)] revealed a significant increase of questions asked from the mid-point to the end-point time session [F_(1,13) = 4.45; p = 0.05; η² = 0.255], revealing an increased interest in learning further with the serious game.

Relationship Between Pedagogical Efficacy and Efficiency Measures

In order to explore any possible association between learning gains across time, and the type of motivation elicited during play, we have examined the correlations between the different measures.

First, a significant correlation was found between AM_M0−M1 scores and learning gains_M0−M1 (r = −0.62; p < 0.02), which indicates that the children who learned the most from software use at mid-point were those who were the less amotivated.

Second, the EM_M1−M2 scores was negatively correlated with both the learning gain _M0−M2 (r = −0.53; p = 0.05), and learning gains _M1−M2 (r = −0.62; p < 0.02), revealing that the children who learned the most from software use in long-term were those who were the less extrinsically motivated, especially for reward expectation.

KidBreath Adherence and Therapeutic Effects

For child engagement measures (Table 3), pairwise comparisons revealed a significant decrease across long-term study, and more specifically from the mid-point to the end-point time session. This lost engagement indicated a dropout behavior from the second mid-point of the study. Similar results were observed for parents' reports. These observations pinpointed their honesty relative to the KidBreath experience.

TABLE 3

Table 3. Means (standard deviation) scores on therapeutic adherence measures (N = 14).

Also, time effect didn't reach the significance for KidBreath-related expectation and judgment score as well as for IPQ scores.

To complete quantitative measures, qualitative data were transcribed during user tests interviews in the 3 sessions (Table 4).

TABLE 4

Table 4. Most frequent expectations reported by parents during interviews in pre-use, and judgment about system utility in mid-point and end-point sessions.

Discussion

The main results from study 1 are now discussed in terms of pedagogical efficacy, pedagogical efficiency, KidBreath adherence, and therapeutic effect.

Regarding short-term and long-term pedagogical efficacy, 79% of the children showed a significant improvement in learning gains on asthma knowledge score after post-using of KidBreath. For these children, results show a better general knowledge about asthma, and a significantly higher learning gain for asthma than for diabetes—the other chronic illness questionnaire used as control condition. Similar results were found in our previous study with non-asthma children (82%) (Delmas et al., 2018).

The significant decrease of “I don't know” answers across assessment sessions comforted the fact that both children learnt more after using KidBreath and that they did not answer it randomly. Pedagogical efficacy was also supported by the negative correlation between learning gains and asthma general knowledge—scores calculated from the answers to the AKQ pre-use questionnaire. This points out that the tool's use benefitted more to children who initially knew less about the illness. This is reinforced by the negative correlation found between learning gains and children expectation in adherence measurement (r = −0.61; p = 0.02), where the tool's use benefitted more for children who not only initially knew less about the illness, but also expected the less about the system to help them for better managing asthma. To check if children didn't answer in a random way, the positive correlation with the answer “I don't know” in AKQ pre-use and the associated relative learning gains showed the tool's use benefitted more to children who were the most aware about their lack of the knowledge asthma-related, i.e., who knew it is in their interest to use KidBreath.

The achieved performance on AKQ at mid- and end-points were similar (M₁ = 71.7%; M₂ = 74.8%) revealing that the asthma-related knowledge was maintained across sessions. Indeed, although young patients completed different questions, the correction component explained most of the time via similar information “blocks” essential to understand for better managing asthma. Hence, children often learned similar information, as up-to-date knowledge, which can explain why the AKQ scores were maintained over time. Although learning activities were imposed in predefined sequences defined by the research team, these near and far results showed KidBreath use triggered strong pedagogical efficacy for learning asthma.

Concerning pedagogical efficiency, a focus was made on identifying which type of motivation (AM, EM, or IM) such a learning tool could generate. This information is essential to guaranty a continued use of the tool in the short to medium term. Mostly used in education (Vallerand et al., 1997), it is rarely used in the development of TEL devices (Cordova and Lepper, 1996), and no mention of its use was made in current serious games. After 1 month-use, a strong intrinsic motivation was scored compared with the others, which matches with our expectations in the previous study (Delmas et al., 2018). Furthermore, the intrinsic motivation stayed constant over the 2-month period and significantly higher than the extrinsic motivation and amotivation. The “ceiling” effect recorded in the end-point for asthma-related knowledge and for intrinsic motivation was found in similar studies carried out on serious games (Brown et al., 1997; Rikkers-Mutsaerts et al., 2012). Indeed, the first month matches with the “discovery” step, where all activities bring enjoyment because of the novelty aspect, but this effect was attenuated in long-term use. This tendency is supported in children playing with a serious game. In an interesting way, the curiosity indicator revealed a significant increase of asthma-related questions asked between the mid- and the end-point assessment time, suggesting some interest in learning further with the serious game.

Concerning therapeutic effect, there is no effect in short or long-term on illness perception. Among the possible explanation ways, two main ones can be advanced. First, a 2-months intervention is not long enough for a deep change in disease perceptions. It is well-known that illness beliefs are resilient and only long-term interventions are able to durably change disease representations of person (Chan et al., 2007; Bindels et al., 2008; Rikkers-Mutsaerts et al., 2012). Second, the age of some children could explain this absence of effect. Indeed, 6 children were born in 2006 (11 years) and 2007 (10 years), which matches the pre-teens period and all the major issues arising for coping, with a strong denial of the disease (de Benedictis and Bush, 2007).

A similar result to the therapeutic effect was found for judgment system efficacy in disease self-management (KidBreath adherence questionnaire). To support the denial hypothesis for coping explained above, most of the children were recruited in the health education association and by pediatrician who worked in this structure, so children initially had some knowledge units. Moreover, this study was composed of children with mild-asthma, so they didn't need to apply information learnt in daily-life situations. Probably, the impact of KidBreath on asthma self-management could be increased in children with severe asthma. Therefore, initial asthma knowledge exhibited by children could be a critical factor for the KidBreath adherence, and then its impact.

The significant decrease of the children cooperation in KidBreath use reported by the parents or themselves between M₁ and M₂ sessions showed a progressive disinterest with time. Once again, initial asthma knowledge in children could be a critical factor for KidBreath use on the long term. Moreover, real-life situations such as family constraints, hospitalizations and parent's implication could also increase KidBreath dropout. For example, participants told us a lack of time for weekly playing KidBreath due to extra-curricular activities, holidays and the homework. However, verbatim also indicated an improvement of child—parent communication on asthma, which is the first step for better managing asthma (Lieberman, 2001). It could be interesting to study the disease-management benefits from the addition of a learning component for parents in KidBreath.

Overall results lead us to develop personalization features into the learning process provided by KidBreath as a way of its improvement. As we noted several times, the variety of children's prior knowledge on asthma could be a key factor for adhering to KidBreath activities, and then supply an actual benefit. Each learner has an initial set of knowledge and skills, and their own learning rate. ITSs were created in the TEL community with optimized learning algorithms offering beneficial personalized learning experiences to students. The study 2 proposed to explore such an improvement of KidBreath.

Study 2: Exploring ITS for Augmenting KidBreath

Related Work

The current advances of learning and teaching technologies have the potential to improve education accessibility and personalization (Dunlosky et al., 2013; Roediger, 2013). ITS have been proposed for these goals, and as a way to provide useful objective metrics on learning (Anderson et al., 1995). One of the most important goals of an ITS is to provide feedback tailored to the learner unique need.

Recently, ITS were implemented in health context such as training medical students in Integrated Management of the Childhood Illness (ICMI) strategy (Muñoz et al., 2010), and for medical problem-based learning in the area of head injury diagnosis (Suebnukarn and Haddawy, 2004); for population gist comprehension of topics relevant to decisions about BRCA (BReast CAncer genes) testing (Wolfe et al., 2015).

One of the main aspects of ITS is to be able to identify the current level of students and how to address particular difficulties in the student learning process. The goal is to be able to choose online the activity that better addresses the challenges being encountered by each particular student. Even two students with the same knowledge will require different activities to progress further due to their previous experience, cognitive skills or preferences. This is a difficult challenge because as ITSs are encountering the students for the first time, it is difficult to know what is the impact of each activity on their progress. There are several approaches to automatically choose exercises based on the current knowledge level of students. We are here particularly interested in optimization methods that rely on minimal prior assumptions about the students or the knowledge domain.

One option already explored is the use of a partial-observable Markov decision process (POMDP) (Rafferty et al., 2011). PODMPs offer an appealing theoretical framework that guarantees an optimal long-term solution for a planning problem. However, in general, as the computational complexity is high, it is practically impossible to find an exact solution to the problem. Some approximate solutions in the domain of ITS have considered the use of aggregations of states instead of tracking the full knowledge components. Another drawback is that POMDPs require a precise student model for which the policy is optimized. If the real student encountered deviates from this model, then the optimality properties are lost.

As noted, before optimizing the sequence of exercises it is important to have some knowledge about the impact of a given exercise in the learning of the KCs, and also to be able to track what each student already masters. A large part of ITS research has been on the modeling aspects of the cognitive and student models. A seminal work on this topic was the Knowledge Tracing framework (Corbett and Anderson, 1994) which builds a detailed cognitive model of the student, of its learning processes by considering a set of independent KCs, the probability of learning them and the probability of correct or wrong answer in exercises that relies on those KCs. More recent methods extend this framework to a Bayesian probabilistic approach (Villano, 1992; Chang et al., 2006) improving the performance and understanding of those methods. Recent methods have started to consider how to learn such models, and variants of it, allowing to simultaneously discover the relation between activities and KC (e.g., Cen et al., 2006; Baker et al., 2008; González-Brenes and Mostow, 2012; González-Brenes et al., 2014).

A more recent approach is to use the Multi-Arm Bandit (MAB) framework (Auer et al., 2002) to manage pedagogical activities (Clement et al., 2015). MABs have the advantage of being extremely computationally efficient and rely on very weak student models. The main drawback is that there is no long-term planning of the best sequence of activities relying on an exploration-exploitation tradeoff to find the best path. Aware of such problem, authors of one such algorithm considered that standard MAB needs to be complemented with a weakly specified knowledge/activity graph to provide a long-term view on the optimization (Clement et al., 2015).

In this work, they developed and tested a set of experiment in simulation to tailor hyperparameters of the algorithm implemented, named ZPDES (Zone of Proximal Development and Empirical Success) and with 400 students to see the impact in a real controlled environment. They showed their method was efficient to personalize and adapt pedagogical content depending of the student progression. Indeed, the students were able to do more different activities and reached them faster with the algorithms than with a predefined sequence defined by a pedagogical expert. As this ITS has been successful, its integration into KidBreath is appeared as a relevant improvement way for providing an automatized personalization.

To test the ITS-augmented KidBreath, it was experimented on similar population (asthma children aged from 8 to 12) and with similar assessment procedure (pedagogical efficacy, efficiency, KidBreath adherence and therapeutic effect) than in study 1.

Material (Details in the Supplementary Material, Appendix 2)

To manage activities, ZPDES combines three things. It uses a hierarchical structure of multiple layers of MABs to explore an activity graph. An activate/deactivate mechanism is present to manage the Zone of Proximal Development explore by the MAB for activities presenting multiple difficulty levels. And an evaluation of the learning progress is used to exploit the activities presenting the best learning interest for the user.

Figure 6 shows the formal description of ZPDES in the algorithm 1 and also how ZPDES works for KidBreath on the diagram. There are three layers in the MAB hierarchy for KidBreath: Theme—Theme's level—Type. The Zone of Proximal Development is represented with yellow outline grouping the active activities.

FIGURE 6

Figure 6. ZPDES exploration of a KidBreath activity graph.

First, the main MAB draws the theme for the next exercise proportionally to the learning progress the student has for each theme. All the Themes are always active, there is no difficulty relation between them and they are all interesting from the start. Each Theme's interest evolves for each children depending of his learning progress for each Theme. For example, in the figure, the Theme T is drawn.

Second, the difficulty Level relative to the theme previously picked is then drawn by active activities. Here, only Level 1 and 2 are active for the theme T. The Level will be stochastically drawn depending of the current learning progress of the children for each Level. In the example, the Level 2 for the Theme T is drawn.

Third, the Type of activity is drawn. As for the Theme, all the Types are always active, there is no difficulty relation between them and all the types are interesting from the start to the end. Also depending of the student learning progress for each Type, each Type's interest will evolve. In this example, the Type Quiz is drawn which leads to the exercise: Theme T; Level 2; Type Quiz.

The ZPDES algorithm was implemented in the “My journey” unit of KidBreath. Therefore, the improved KidBreath version is unchanged except for this learning unit. Precisely, in “My journey,” the framework within a learning activity didn't change, but the only difference was in the display sequence of learning activities in “My journey,” which take into account answers provided by children in quiz, MCQ or games to go to the next activity, thanks to the integration of ZPDES algorithm. As in the previous study, the child could switch to other units, but if they wanted to return to play in “My journey,” the learning activity in progress was also displayed and they had to complete it to go to the next one.

Method

The method used was very close to the one used for the study 1. So, next sections document differences between the two studies rather than similarities.