Multiple Negative Emotions During Learning With Digital Learning Environments – Evidence on Their Detrimental Effect on Learning From Two Methodological Approaches

Emotions are a core factor of learning. Studies have shown that multiple emotions are co-experienced during learning and have a significant impact on learning outcomes. The present study investigated the importance of multiple, co-occurring emotions during learning about human biology with MetaTutor, a hypermedia-based tutoring system. Person-centered as well as variable-centered approaches of cluster analyses were used to identify emotion clusters. The person-centered clustering analyses indicated three emotion profiles: a positive, negative and neutral profile. Students with a negative profile learned less than those with other profiles and also reported less usage of emotion regulation strategies. Emotion patterns identified through spectral co-clustering confirmed these results. Throughout the learning activity, emotions built a stable correlational structure of a positive, a negative, a neutral and a boredom emotion pattern. Positive emotion pattern scores before the learning activity and negative emotion pattern scores during the learning activity predicted learning, but not consistently. These results reveal the importance of negative emotions during learning with MetaTutor. Potential moderating factors and implications for the design and development of educational interventions that target emotions and emotion regulation with digital learning environments are discussed.

Introduction

Learning is a complex multi-faceted process that requires students to deploy, monitor, and regulate their cognitive, metacognitive, affective and motivational processes based on the learning environment and the learning task and goal (Azevedo et al., 2018). Emotions play a central role in this context. They significantly impact and drive processes that are quintessential to learning, such as attention, perception, memory (Lewis et al., 2008; Tyng et al., 2017), and metacognition (Azevedo et al., 2017). Furthermore, a long tradition of research has shown that emotions are directly related to learning outcomes and academic achievement (Boekaerts and Pekrun, 2015). Even though initial investigations on emotions and learning has almost exclusively focused on the importance of anxiety in learning and test situations (Pekrun et al., 2002), research on emotions and learning has diverged into investigations of a broad variety of affective states and emotions in differing learning contexts (e.g., classroom settings, research with advanced learning technologies or informal learning settings; Azevedo et al., 2019). These studies have demonstrated that many different emotions are commonly experienced in learning settings (e.g., boredom, confusion, or frustration; D’Mello, 2013) and they have a significant impact on students’ performance (e.g., Pekrun et al., 2002; D’Mello et al., 2014). However, some important aspects of emotional experiences still have not been extensively researched in learning contexts. For example, most of the research in this context, particularly research during learning with digital learning environments, focused on the importance of single discrete emotions or sets of discrete emotions using variable-centered approaches. Research investigating emotions in other contexts, on the other hand, has revealed that approaches that consider multiple emotions simultaneously show great promise (e.g., Fortunato and Goldblatt, 2006; Vansteenkiste et al., 2009). Only a few studies have investigated the complexity of students’ (co-occurring) emotional experiences during learning using person-centered approaches (Ganotice et al., 2016; Jarrell et al., 2016, 2017; Robinson et al., 2017; Sinclair et al., 2018). These studies have found that groups of students who differ in their emotional experiences during learning in regard to multiple emotions (so called emotion profiles) also meaningfully differ in their learning outcomes and academic achievement. The goal of this study was to combine person- and variable-centered approaches to examining emotions during learning with a digital learning environment. We extended upon previous research by considering a broader range of emotion measures than previous studies (i.e., academic achievement emotions and learning-centered emotions), incorporating emotion regulation and temporal dynamics of emotions, and by substantiating person-centered analyses with a novel variable-centered approach.

Emotions During Learning With Digital Learning Environments

Emotions are an essential component of learning activities across settings. Students’ emotional experiences when learning with technologies are diverse, have been investigated on the basis of several frameworks (D’Mello, 2013), and have been classified in various categories, including academic achievement emotions (Pekrun et al., 2002), epistemic or learning-centered emotions (D’Mello and Graesser, 2012; Pekrun et al., 2017b), and basic emotions (Ekman and Friesen, 1971; Ekman, 1992). Pekrun et al. (2002) and Pekrun (2006) academic achievement emotions approach distinguishes academic emotions differing in their valence (positive vs. negative) and the perceived level of control by the learner, including enjoyment (positive and high control), anxiety (negative and medium control), and hopelessness (negative and low control). Learning-centered emotions approaches (also referred to as cognitive affective states or epistemic emotions; D’Mello and Graesser, 2012; Muis et al., 2015; Pekrun et al., 2017b) focus on emotions that are directly related to knowledge-generating aspects of cognitive processes (e.g., overcoming impasses during learning), including boredom, confusion and frustration. According to Ekman (1992) six basic emotions can be distinguished across cultural contexts and reliably identified from facial expressions, including anger, happiness, and surprise. An extensive amount of research has shown that emotions significantly impact learning processes, outcomes, and academic achievement (Pekrun and Linnenbrink-Garcia, 2014). The majority of studies revealed that the way emotions impact learning and achievement is closely related to their valence. More specifically, positive emotions are positively, and negative emotions are negatively related to the learning process and learning outcomes (e.g., Pekrun et al., 2002, 2017a; Pekrun and Linnenbrink-Garcia, 2012). However, there is also evidence opposing this general pattern. For example, studies identified detrimental effects of positive emotions on the accuracy of metacognitive judgments creating an illusion of learning (Baumeister et al., 2015). Negative emotions on the other hand were positively associated with learning when they triggered deep processing of contents and were resolved by the students in a timely manner (see below, e.g., D’Mello and Graesser, 2014). This state of research indicates that, despite the overall tendency of beneficial effects of positive emotions and detrimental effects of negative emotions, further factors need to be considered to predict and explain the effects of emotions during learning.

A particular branch of research investigates (self-regulated) learning processes when learning with digital learning environments (Gegenfurtner et al., 2019), including hypermedia learning environments (e.g., Opfermann et al., 2013), intelligent tutoring systems (e.g., Azevedo et al., 2016; Harley et al., 2017), and game-based learning environments (e.g., Sabourin and Lester, 2014; Taub et al., 2018). These learning technologies have been designed and implemented to foster student learning about specific topics and have been shown to meaningfully enhance learning (Zheng, 2016). Digital learning environments include specific affordances that are directly linked to students’ emotions. For example, research has demonstrated that the design of digital learning environments (e.g., shapes and colors; Plass et al., 2014), their structure (e.g., complex, non-linear structure; Arguel et al., 2019), and scaffolds incorporated in such systems (e.g., prompts and feedback by pedagogical agents; Harley et al., 2017) can impact students’ emotions. More specifically, digital learning environments can elicit and alter emotional processes or assist the learner in regulating them and provide unique opportunities for research to investigate emotions in ways hardly achievable in other contexts. For instance, multi-channel trace data can be collected with digital learning environments to measure emotions with minimal interruptions to the learning process (e.g., through automated detection of facial expressions; D’Mello, 2017; Azevedo et al., 2019). The dynamics of affective states model is a prominent theoretical framework in this line of research that focuses on the dynamic unfolding of specific, learning-centered emotions (D’Mello and Graesser, 2012)¹. More specifically, D’Mello and Graesser (2012) posited that confusion is elicited by impasses encountered during complex learning processes. This confusion can be beneficial to learning when it can be resolved, and the impasse can be overcome. Prolonged experiences of confusion on the other hand is theorized to lead to frustration and eventually boredom, which ultimately lead to disengagement and poor learning outcomes. Given that digital learning environments challenge students with learning tasks that require to develop a deep understanding of science concepts, or a solution for complex problems, such impasses are particularly likely to occur when learning with these systems. D’Mello and Graesser (2014) found a positive relation between (partially) resolved confusion and learning in a problem-solving task and a scientific reasoning task in an intelligent tutoring system. Another study by Taub et al. (2019) furthermore showed that the experience of frustration was linked to higher accuracy in the use of cognitive learning strategies (i.e., note-taking) with MetaTutor. However, they did not find a significant relation between emotions and learning gain.

Other studies on emotions and learning with digital learning environments (e.g., intelligent tutoring systems and game-based learning environments) on the other hand found detrimental effects of negative emotions. Initial studies on the relation between emotions and learning in AutoTutor identified significant detrimental effects of boredom for learning (Craig et al., 2004; Graesser et al., 2008). Across three studies using different digital learning environments, Baker et al. (2010) found further support for these findings by showing that boredom was the most persistent emotion (i.e., students were unlikely to transition from boredom to another emotion), and that boredom was the only emotion to be associated with maladaptive behaviors (i.e., gaming the system). Sabourin and Lester (2014) identified a positive relation between positive emotions and learning gains. Furthermore, they observed a negative association of confusion and boredom with learning gains in a game-based learning environment. A study by Grafsgaard et al. (2014) revealed that indicators of facially expressed frustration were negatively predictive of learning gain.

Taken together, these studies demonstrated the importance of learning-centered emotions during learning with digital learning environments (for a recent review see Arguel et al., 2019). However, they also demonstrated a profoundly controversial relation between (negative) emotions and learning. This clearly indicates further research is needed to disentangle the manifold relation between emotions, learning, and learning outcomes by identifying factors that explain these contradictory relations. One such factor that has been rarely considered in the aforementioned studies on emotions in digital learning environments is the co-occurrence of emotions. Even though studies have shown that the emotions outlined above have differential effects on learning depending on other affective states they are accompanied by or lead to (e.g., D’Mello and Graesser, 2012; Goetz et al., 2014; Riemer and Schrader, 2019), the co-occurrence of emotions and the breadth of emotional experiences has rarely been considered in this context.

Person Centered Approaches to Emotions

Research on emotions during self-regulated learning has indicated that a variety of emotional states and processes impact learning and performance in meaningful ways. While these studies have greatly contributed to a comprehensive understanding of emotions in learning situations, especially when learning with digital learning environments, they have not fully considered the breadth of emotional experience of an individual. More specifically, the variable-centered approach used by these studies focuses on singular emotional states or a pre-selected set of emotions while controlling for the impact of other emotions. Emotion research on the other hand suggests that individuals can experience multiple emotions concurrently, and that these emotions affect each other reciprocally, which ultimately impacts thoughts and behaviors (e.g., Lazarus, 2006; Fernando et al., 2014). Person-centered approaches typically identify groups of students with similar emotional experiences in regard to multiple emotions at a certain point of time (often referred to as emotion profiles). These profiles are then compared to another and related to relevant outcome measures (e.g., learning and academic achievement). For example, multi-level investigations of affect in college students have revealed that spurs of negative emotions coupled with positive trait affectivity are associated with greater academic growth than positive or negative affect alone (Barker et al., 2016). Furthermore, the added value of this approach has been repetitively shown outside of educational contexts (e.g., Vansteenkiste et al., 2009; Fernando et al., 2014). In research in education settings, this approach is still quite rare. We identified five studies that used a person-centered analytical approach in different educational contexts (see Table 1 for a brief overview).

TABLE 1

Table 1. Overview of person-centered studies on emotions during learning.

Jarrell et al. (2016, 2017) investigated emotions when learning with a computer-based learning environment using a person-centered approach in two studies. Five discrete emotional states (enjoyment, pride, hope, shame, and anger) measured with the Achievement Emotions Questionnaire (AEQ; Pekrun et al., 2002) were used to cluster students with similar emotional experiences. In both studies, a three-profile solution including a positive, negative, and low emotional experience profile, was identified. These profiles were subsequently related to learning outcomes. The first study (N = 26) revealed no significant differences in performance between profiles. In the follow-up study (N = 30) Jarrell et al. (2017) investigated differences in diagnostic performance efficiency between emotion profiles. They found that the negative emotion profile was outperformed by at least one other profile averaged across levels of difficulty (easy, medium, hard) and for easy and hard tasks, but not for tasks with medium difficulty.

Further investigations of emotions through a person-centered approach were conducted by Ganotice et al. (2016) in two secondary school samples. Similar to the studies outlined above, discrete emotional states (enjoyment, hope, pride, anger, anxiety, shame, hopelessness, boredom) measured through the AEQ (Pekrun et al., 2002) were used for clustering. In a domain general or a math-specific context, four emotion profiles were identified. These profiles included a high positive and high shame profile, a moderate positive and negative emotion profile, a high negative emotion profile, and a high positive emotion profile. These profiles were compared in regard to school engagement, motivation, and math performance. Results showed that profiles with high positive emotions were the most adaptive profiles while the high negative emotion profile was the least adaptive.

Robinson et al. (2017) investigated affective profiles in an undergraduate anatomy course. Other than previous person-centered studies, this research used two dimensions of affect (positive/negative × activated/deactivated, see Ben-Eliyahu and Linnenbrink-Garcia, 2013) as clustering variables. Through a two-step procedure, they identified four emotion profiles including a positive, a deactivated, a negative, and a moderate negative profile. Comparison in academic achievement revealed that the deactivated profile showed higher academic achievement than both negative profiles (negative and moderate negative) throughout three exams. Robinson et al. (2017) also found differences between the positive and the negative profile, but not throughout all exams. Lastly, they investigated the mediating role of (dis-) engagement and found that higher levels of performance for the positive and deactivated profile were mediated through lower levels of disengagement.

Lastly, Sinclair et al. (2018) investigated emotion profiles displayed in an undergraduate student sample that learned about the human circulatory system using MetaTutor (see 5.3 MetaTutor). They used five discrete emotion states (enjoyment, curiosity, pride, boredom, and frustration) measured at five time points before and during learning using latent profile analysis. Similar to the studies above, they found a positive, negative (bored/frustrated), and moderate emotion profile. Subsequently they investigated transitions between profiles and found that students from the negative profile were least likely to transition to another profile. Lastly, they found that learning gain predicted the transitions between profiles at specific, selected time points.

Taken together, these studies demonstrate that a person-centered approach can reveal emotion profiles across contexts, ranging from laboratory studies to research in schools and university. Furthermore, all studies have found that these profiles are significantly related to performance, academic achievement, and related constructs. Most of the previous studies have not incorporated learning-centered or epistemic emotions (e.g. boredom, confusion, and frustration; D’Mello and Graesser, 2012). On the other hand, previous research on emotions when learning with digital learning environments has found that these emotions significantly impact learning in varying ways. The finding that these emotions can have a positive or negative impact on learning is particularly interesting for person-centered research as the contradicting implications might be explained though co-occurring emotions (i.e., profiles that show similar levels of confusion or frustration, but varying levels of other emotions). The only study that investigated learning-centered emotions (Sinclair et al., 2018) on the other hand did not consider achievement emotions in their analysis, which makes comparisons across studies difficult. We aim to address this issue by including learning-centered emotions in addition to academic achievement emotions that were used in most of the person-centered studies outlined so far.

Furthermore, the aforementioned studies have investigated different constructs related to emotions and performance such as motivation (Ganotice et al., 2016) or engagement (Robinson et al., 2017) to substantiate their findings. None of the studies investigated the role of emotion regulation in this context. Emotion regulation is an essential component to emotional experiences in learning contexts and is a critical link between emotional experience and academic outcomes (Gross, 2015; Harley et al., 2019). It describes students’ efforts to influence which emotions they experience, when they experience these emotions and how they express them (Harley et al., 2019). Emotion regulation strategies are for example the cognitive reappraisal of emotional experiences or modification of the situation that elicited the emotion (Gross, 2015). Spann et al. (2019) found that emotion regulation significantly influenced the relation between emotions and learning in a game-based learning environment. More specifically, they found that cognitive reappraisal led to higher learning outcomes for highly confused, frustrated, and engaged students, but was not as effective for students with low levels of confusion, frustration and engagement. Incorporating emotion regulation could shed light on the development of emotions in relation to specific profiles. Adaptive profiles (such as described by Ganotice et al., 2016) are potentially defined by higher levels of emotion regulation to cope with high levels of negative emotions. To investigate this subject matter, temporal investigations of emotions related to emotion profiles similar to Sinclair’s approach (Sinclair et al., 2018) are necessary. This includes, investigating the self-reported use of emotion regulation strategies for the different emotion profiles, and exploring to what extent the intensity of emotional experiences fluctuates over time within profiles.

Lastly, the studies outlined above were limited to using person-centered approaches only. While the great value of this type of research has been shown, we argue that supplementing person-centered with other approaches can be essential to their understanding. More specifically, identifying if the distinguishing characteristics of profiles (e.g., varying levels of positive or negative emotion intensity) can be replicated through variable-centered approaches can provide additional insight on the origin of these profiles. Such approaches could differentiate if profiles are based on natural co-occurrence of emotions (e.g., high correlations between negative emotions) or specific combinations of individual emotional experiences (e.g., a profile with high levels of boredom and other negative emotions versus a profile with high levels of boredom and low levels of other negative emotions). Furthermore, replicating results using two different methodologies would reveal their level of robustness, which is particularly important in this context, because emotion profiles are identified through data driven approaches (guided by previous research).

Current Study

The current study aims to address the issues outlined above by identifying emotion profiles of students who learned with MetaTutor and relate them to learning outcomes. To this end we decided to adapt the person-centered analytical procedure outlined by Vansteenkiste et al. (2009) and Robinson et al. (2017) for the identification of emotion profiles. Additionally, we demonstrate how a variable-centered approach can substantiate these results by relating patterns of emotions to emotion profiles and learning outcomes throughout different phases of learning (i.e., before the learning phase, at the start of the learning phase, and at the end of the learning phase, see section Emotion Items). More specifically, we aim to answer the following questions.

1.1 Which emotion profiles can be identified during SRL with MetaTutor and how can they be described? Given that the specific profiles are highly dependent on the number of clusters, no specific hypothesis can be formulated a priori. However, based on previous person-centered studies, we expect a negative and positive emotion profile (see Ganotice et al., 2016; Jarrell et al., 2016, 2017; Robinson et al., 2017; Sinclair et al., 2018). Additionally, further likely profiles can include a low-intensity or moderate intensity profile for all emotions.

1.2 Are there significant differences in learning outcomes between the profiles? Based on previous research, we expect the profile with the highest values of negative emotions to display the lowest learning gain (Ganotice et al., 2016; Jarrell et al., 2016, 2017; Robinson et al., 2017).

1.3 Are there significant differences in self-reported use of habitual emotion regulation strategies between the profiles? Based on research on emotion regulation, we expect profiles characterized by high negative emotion intensities to indicate lower levels of self-reported use of emotion regulation strategies (Harley et al., 2019).

2.1 How can stable patterns of emotions can be identified throughout the different phases of the learning session and how can they be described? Similar to our first research question, we expect a strong differentiation between negative and positive emotions in the different phases. Additionally, a strong differentiation between activating and deactivating emotions is expected (Ben-Eliyahu and Linnenbrink-Garcia, 2013). Furthermore, because neutral – per definition – refers to the absence of perceivable and detectable emotions, we hypothesize neutral to represent its own cluster (potentially paired with emotions that show low intensities overall). Lastly, based on the reoccurring finding that specific emotions are positively and/or negatively related to learning, we expect boredom, confusion or frustration to form separate cluster(s) from other negative emotions (e.g., D’Mello and Graesser, 2012).

2.2 How are emotion profiles related to the phase-specific patterns of emotions? We expect emotion profiles to significantly differ in regard to emotion clusters that are defined by valence as all previous studies included profiles that were defined by positive and negative emotions (Ganotice et al., 2016; Jarrell et al., 2016, 2017; Robinson et al., 2017). In an exploratory step we will investigate if these differences are stable over time or if they arise throughout specific parts of the learning session.

2.3 How can the phase-specific patterns of emotions predict learning outcomes in the respective phases of the learning activity? Based on previous research, we expect negative emotions to be most predictive of learning. However, the direction of this interaction will be explored, as previous research has shown controversial results in this regard.

Materials and Methods

Participants

One hundred ninety-four (N = 194) undergraduate students (aged between 18 and 41, M = 20.46 years, SD = 2.96 years; 53% female) from three large public North American universities participated in a 2-day laboratory study. They were randomly assigned either to the prompt and feedback (P + F) or control (C) condition (see section MetaTutor), and monetarily compensated for their time ($10 per hour, up to $40). For the present study, only participants that filled out a sufficient number of emotion questionnaires (see section Emotion Items) were included in analyses, resulting in a sample size of one hundred seventy-six (N = 176) students.

Procedure

The experiment was conducted over 2 days. On the first day, participants signed a consent form, filled in demographics questions, and completed several self-report measures (e.g., the Achievement Emotions Questionnaire – Pekrun et al., 2002 and the Emotion Regulation Questionnaire – Gross and John, 2003). Lastly, after responding to the questionnaires, participants took a 30-item pretest about the human circulatory system.

On the second day of the experiment, students were first introduced to the learning task and learning environment. They were instructed to set two learning sub goals before the beginning of the learning phase. During the learning phase, participants had to engage in self-regulated learning by reading texts, inspecting corresponding diagrams, and completing quizzes. Moreover, regardless of the experimental condition (see section MetaTutor) students were free to indicate their use of certain cognitive (e.g., note taking) or metacognitive learning strategies and activities using the SRL palette implemented in MetaTutor’s interface (see section MetaTutor). Additionally, quizzes and self-report measures (i.e., the emotion and values [EV] questionnaire; Azevedo et al., 2013) were administered based on specific rules implemented by the system (e.g., the EV was conducted on a time-based threshold – roughly every 14 min during the learning session with MetaTutor).

After the 60-min learning phase, students were directed to the post test (i.e., 30-item test about the circulatory system) and had to complete a last set of self-reports (e.g., an EV directly before the posttest) before they were debriefed by the research assistant.

During the experiment, multiple channels of multimodal data, including eye tracking, galvanic skin response, and automated analysis of facial expressions were collected. However, these process measures were not analyzed in the present study.

MetaTutor

MetaTutor is a hypermedia-based tutoring system that fosters self-regulated learning processes while learning about the human circulatory system (Azevedo et al., 2018). The system was designed using a set of production rules, which fire based on how students monitor and control their understanding of the text and relevancy of the current page to the sub-goal they are working on. In addition to the processes being prompted by the pedagogical agents based on the production rules, participants were able to engage in any process of their choice. The MetaTutor learning environment was strategically designed to foster the use of cognitive learning strategies and metacognitive monitoring processes (see Figure 1). For example, a timer (A) and sub goal progress bar (C) allow students to monitor their progress toward achieving their sub goals and overall learning goal. The table of contents (B) provides students all the content page titles so they can select the appropriate pages to read for achieving their sub goals. There are seven pre-set sub goals in the environment (path of blood flow, heartbeat, heart components, blood vessels, blood components, purposes of the circulatory system, and malfunctions of the circulatory system). Prior to the 60-min learning session, students are progressed through a sub goal setting phase where they are guided to set two of those sub goals. The content text (D) and diagram (E) facilitate knowledge acquisition and foster coordinating information between the text and diagram. The SRL palette (F) provides students the opportunity to select cognitive learning strategies (i.e., prior knowledge activation, take notes, summarize, make an inference) and metacognitive monitoring processes (judgment of learning, feeling of knowing, content evaluation) they want to use during learning about the human circulatory system.

FIGURE 1

Figure 1. Screenshot of the MetaTutor interface. (A) Timer, (B) Table of contents, (C) Progress bar, (D) Content text, (E) Content image, (F) SRL palette, and (G) The pedagogical agent.

There are four pedagogical agents with one present at a time (G), where each agent focuses on a specific component of SRL. Gavin (shown in Figure 1) guides students through the learning environment and administers self-report questionnaires. Pam fosters planning by helping students set sub goals and activate their prior knowledge. Sam focuses on strategy use. Mary emphasizes monitoring processes. The amount of assistance provided by the pedagogical agents depends on the experimental condition students are assigned to. In the P + F condition, the agents prompt students to engage in SRL processes (using time- and event-based production rules). They also provide feedback on how they performed. For example, Sam will prompt students to make a summary, and once they have done so, he will tell them it is too long, too short, acceptable, etc. In the C condition, students are not prompted by the agents, nor are they given any feedback on their performance. In this condition, students can still initiate the use of cognitive and metacognitive processes, however, they are still not given any feedback, whereas in the P + F condition, students can also self-initiate the use of these processes, and will be given feedback on their performance.

Measures

Emotion Items

Students’ emotional experiences at the start, during, and at the end of the learning session were measured using the Emotion-Values Questionnaire (EV; Azevedo et al., 2013). The EV covers 15 emotional states as well as two questions asking about the perceived value and the students’ ability to perform well on the current task on a five-point Likert-scale (ranging from 1 – “Strongly Disagree” to 5 – “Strongly Agree”). Additionally, two forced choice items asked the participants to select the emotion that best describes how they currently feel out of 15 (all emotional states from the EV) and 7 options (basic emotions), respectively. The emotional states included in the EV were based on extensive research on achievement emotions in academic settings (Pekrun et al., 2002; Pekrun, 2006), as well as on research on learning-centered emotions/epistemic emotions (e.g., D’Mello and Graesser, 2012; Muis et al., 2015; Pekrun et al., 2017b). The questionnaire covers the following emotions (in order of administration): enjoyment, hope, pride, frustration, anxiety, shame, hopelessness, boredom, surprise, contempt, confusion, curiosity, sadness, eureka, and neutral. A definition and an example were provided for each emotional state during each administration.

The EV was administered at fixed points of time before and after the learning phase, and time-based during the learning phase. More specifically, the questionnaire was administered directly before and after participants set their learning sub goals, and before the actual learning phase. During the learning activity the questionnaire was administered every 14 min. Lastly, the final EV was administered when the learning phase was finished, directly before the post test. The number of EVs completed varied between participants because the administration during the learning phase was postponed when key learning activities took place. In particular, the questionnaire did not interrupt any of the user- or agent-initiated learning strategies that required completing quizzes or filling out questionnaires. For example, if a student initiated the sequence of finishing the current learning sub goal, they had to fill out a 10-item multiple-choice quiz on the current topic and received feedback depending on the experimental condition (see section MetaTutor). If an EV should have been administered during that sequence, it was postponed until the end of the sequence, potentially delaying it by several minutes. This resulted in a range of four to eight EVs completed between participants. To allow for comparisons of participants, we decided to limit the EVs analyzed in the present study to six points of time relative to the start and the end of the learning session. Therefore, only participants that completed at least six EVs were considered for analyses, resulting in a final sample size of one-hundred-seventy-six students (N = 176). The following EVs were selected: (1) the first two EVs that were completed at the beginning and end of the sub goal setting phase, (2) the third and fourth EV, which took place in the first half of the learning phase, and (3) the last two EVs, which was the last questionnaire presented during the learning phase, and the final EV immediately prior to the post test. Due to missing data, “Eureka” was excluded from analyses in the current study, yielding 14 discrete emotions considered for analyses.

Pre and Post Tests

Prior knowledge and learning outcomes were measured using two 30-item multiple choice tests covering conceptual knowledge of the human circulatory system. The measures were developed by a domain expert in the subject matter. Each question had four potential answers and one correct solution. The order of two equivalent versions of the tests was randomized and counterbalanced across experimental conditions. Percent correct for both measures were computed for analyses.

Emotion Regulation Questionnaire

Students’ self-reported habitual use of emotion regulation strategies was measured using the emotion regulation questionnaire (ERQ; Gross and John, 2003). The 10-item questionnaire features two sub-scales asking about the use of emotion regulation strategies using a seven-point Likert scale (ranging from 1 – strongly disagree to 7 – strongly agree). More specially, mean values for the sub-scales expressive suppression (4 items, α = 0.78; e.g., “I keep my emotions to myself.”) and cognitive reappraisal (6 items, α = 0.84; e.g., “I control my emotions by changing the way I think about the situation I’m in.”) were calculated for analyses.

Statistical Analyses

Statistical analyses in the present study were conducted using R (R Core Team, 2019), Python (Van Rossum and Drake, 2011), and SPSS (SPSS, 2012). Before the initial analyses, we investigated if the mean scores for each emotion computed over the six administrations of the EV for clustering contained significant outliers using Grubbs (1969) approach (implemented through the ‘grubbs.test’ function of the outlier package for R; Komsta, 2011). In total, 12 univariate outliers were replaced by the closest non-outlier value (three for shame, one for hopelessness, two for surprise, two for confusion, and five for surprise). Furthermore, investigations of the skewness and kurtosis (values < 2; George and Mallery, 1999) revealed that all of the variables used for analyses (i.e., mean emotion scores, emotion cluster scores and learning measures) were within acceptable ranges of normal distribution.

The person-centered methodological approach for the identification of emotion profiles was based on previous studies investigating affective, emotional, or motivational profiles (Vansteenkiste et al., 2009; Robinson et al., 2017). More specifically, we first used the ‘hclust’ function of R’s stats package to compute a range of profile solutions using Ward’s method and extracted the cluster centroids for each profile. We used agglomeration coefficients obtained through the SPSS classification function (SPSS, 2012), minimum number of profile size (Fernando et al., 2014), and cluster fit indices from ‘Nbclust’ (Charrad et al., 2014) to identify the eligible range of clusters. Subsequently, k-means clustering analysis with these centroids as starting points was conducted (‘kmeans’ function of the ‘stats’ library) to obtain the most distinctive set of profiles. As a last step in the cluster identification we used the cross-validation procedure outlined by Breckenridge (2000) to assess the stability of the solution (using self-implemented function based on the ‘knn’ function of the ‘class’ library; Venables and Ripley, 2002). Together with investigations of explained variance in the clustering variables and redundancy of the clusters, this criterion was used to determine the final cluster solution. Clustering methodology was chosen because the suitability of clustering over other methodological approaches in this context has been repeatedly showcased by previous research (e.g., Robinson et al., 2017).

Subsequently we used a latent growth linear mixed effect model to investigate differences in learning outcomes between emotion profiles. Models were fit using ‘lmer’ from the ‘lme4’ library (Bates et al., 2014). Summary statistics were extracted via the ‘analyze’ function of ‘psycho’ (Makowski, 2018) and post hoc comparisons were conducted using ‘glht’ from ‘multicomp’ (Hothorn et al., 2008). Additionally, this analysis was repeated for all profile solutions (including the initial solutions from hierarchical clustering) to assess if the findings were stable throughout different profile configurations.

Then spectral co-clustering – a machine learning clustering approach – implemented through the ‘SpectralCoclustering’ function of the Python library ‘scikit-learn’ was used to substantiate the relation between emotions and learning identified through the profiling approach (Pedregosa et al., 2011). Specifically, we grouped emotions into clusters based on their correlation across all measurement points and separately for each time point. The emotion cluster solution was selected based on its stability over all administrations of the EV and alignment to previous research. Then, principal component analysis (‘PCA’ function of ‘scikit-learn’) with one main component was used to obtain participants’ scores for each emotion cluster at each measurement point. Additionally, the internal consistency of emotion clusters was assessed though Cronbach’s Alpha (‘alpha’ of R’s ‘psych’ package; Revelle, 2017). The obtained scores were then used in multiple regressions for each time point separately to assess how the emotion clusters are related to learning. Regression weights were calculated using the ‘lm.beta’ function R’s ‘lm.beta’ package (Behrendt, 2014).²

Preliminary Analyses

To control for the potential effect of the experimental manipulation of the present study (i.e., the control and prompt + feedback conditions) on the results described in the following sections, all variables included in the analyses were compared between the experimental conditions using multivariate analyses of variance (MANOVAs). Results showed no systematic differences in pre- and posttest scores, emotion scores, or emotion cluster scores between the conditions (all p > 0.05; except negative emotions cluster scores for EV 1: p < 0.05). Additionally, we conducted chi-square tests for each profile solution to test if the experimental conditions were equally represented in each emotion profile. Results revealed no significant differences in the distribution of experimental conditions for any of the emotion profiles identified.

Person-Centered Approach: Emotion Profiles

Identifying Emotion Profiles

To identify emotion profiles, students with similar self-reported emotional experiences were grouped using a two-step clustering approach. More specifically, first, hierarchical clustering (Ward’s method) was used on the squared Euclidian distance matrix for the mean values of each emotion for each participant throughout all six time points (see above). Each participant started as their own cluster in the hierarchical clustering analyses. Then the closest participants were merged into a cluster. This step was repeated until all participants were merged into a single cluster, resulting in a range of cluster solutions between the number of participants (i.e., each participant as their own cluster) and a singular cluster. To identify the profile solutions eligible for subsequent analyses, we used three criteria: (1) the scree-plot of agglomeration coefficients to identify the point where the addition of clusters did not substantially decrease the agglomeration coefficient, (2) a sufficient profile size for statistical analyses (n > 10; Fernando et al., 2014), and (3) multiple cluster fit indices (Charrad et al., 2014). Agglomeration coefficient indicated that merging a three-cluster solution into two clusters was not practical (Δ_coefficient = 233.93). A second drop in agglomeration coefficients was identified for the addition of a sixth cluster, but was less substantial (Δ_coefficient = 73.379). While this procedure favored solutions with more than six profiles, the second criterion limited the number of profiles to a maximum of seven, as all further profile solutions included profile(s) with less than ten participants. Lastly, we compared the solutions that were sufficient for both criteria in regard to 26 fit indices (see Charrad et al., 2014 for a complete list of the indices) and found equal support for the three to five profile solutions and little to no support for the six and seven profile solutions. Accordingly, the three-, four-, and five-profile solutions were selected for further analyses.³ Preliminary analyses on the structure of the clusters revealed a noteworthy feature. A single emotion profile with higher negative emotion intensities than other profiles (n = 29) was a stable component of all solutions outlined above.

As a second step in the identification of emotion profiles we used k-means clustering, a non-hierarchical clustering procedure, in order to increase similarity within clusters and differences between clusters. More specifically, for the previously selected three- to five-cluster solutions, we first extracted the cluster centroids. These values were then used as starting points of the k-means clustering instead of starting with randomized seeds. In this procedure the number of clusters is defined a priori. Then a starting seed was used as the initial centroid of a cluster and participants that were in proximity to that centroid (measured through a distance threshold) were assigned to that cluster. This procedure was repeated for each starting seed until all participants were assigned to a cluster (Fortunato and Goldblatt, 2006). K-means clustering was chosen because this procedure simultaneously maximizes between cluster distances (i.e., increased differences between emotion profiles) and minimizes within-cluster variance (i.e., increased similarity within profiles; Eshghi et al., 2011). After obtaining the respective cluster solution, we then assessed the rate of agreement between the hierarchical and k-means approaches. Both clustering methods showed sufficient rate agreement (K₃ = 0.76; K₄ = 0.78; K₅ = 0.78). This indicated that the k-means clustering altered the initial profiles obtained through the hierarchical clustering but maintained the overall structure and demonstrates the robustness of the identified profiles. To test if the aggregation of self-reported emotion intensities had a significant impact on the obtained emotion profiles, we re-ran all previous steps using all six measurement points for the fourteen emotions as clustering variables. Comparison of the profiles identified by clustering means and the profiles identified by clustering all measurement points demonstrated high to very high agreement (K₃ = 0.85; K₄ = 0.91; K₅ = 0.88). This indicated that our data supports the use of mean values as clustering variables and further underlined the robustness of the clustering procedure.

To select the emotion profiles for subsequent analyses we first compared the explained variance in mean emotion intensities between the solutions with different numbers of emotion profiles. The three-profile solution explained moderate levels of variance for all mean emotion intensities, except neutral, surprise, anxiety and contempt (see Table 2). The four-profile solution explained more variance for most of the emotions, but also showed lower levels of explained variance for specific emotions (i.e., contempt and confusion). This pattern also applied to the comparison of the four- and five-profile solutions. However, while the four-profile solution added a profile that was primarily defined by boredom in addition to the neutral, positive, and negative emotion profiles of the three-profile solution, the five-profile solution only added a profile that was largely redundant to the positive emotion profile (with higher levels of curiosity, surprise and anxiety). Based on the largely redundant nature of this profile (a criteria used by Fernando et al., 2014), we decided not to consider this solution.

TABLE 2

Table 2. Explained variance by profile-solution.

As the final step for selecting the most suitable cluster solution, we cross validated the three- and four-profile solutions following the procedure outlined by Breckenridge (2000). More specifically, we split our sample randomly into two equally large sub samples. Then, the two-step clustering procedure outlined above was separately applied to each of the sub samples. The two sub samples were subsequently compared with a k-nearest-neighbors approach. More specifically, each participant of a sub sample was assigned to a new cluster value based on their most similar counterparts in the other sub sample (their nearest neighbors). To assess the robustness, Kohen’s Kappa (as a measure for agreement) was calculated based on the initial (obtained through the two-step approach) and new cluster assignment (obtained through the nearest neighbors procedure) in both samples. To increase the robustness of the cross-validation, we repeated this procedure twenty times and averaged Kappa values across all iterations (i.e., 20-fold cross validation). Results indicated that the three-profile solution (K = 0.65) showed sufficient stability (i.e., K > 0.60; Breckenridge, 2000; Asendorpf et al., 2001), but the four-profile solution did not (K = 0.56). Therefore, the three-profile solution was selected as the final profile solution (see Figure 2 for a comparison of mean emotion intensities between the three profiles). Means and standard deviations for mean emotion intensities, and pre and post test scores of the three-profile solution are displayed in Table 3. The three profiles can be described by their most distinct features as follows⁴. The first profile (n = 75) displayed low to moderate levels for all emotions except boredom and neutral, which were at moderate levels. The neutral score was higher than for the other profiles. Accordingly, we refer to this profile as neutral. The second profile (n = 62) showed moderate to high levels for most of the positive emotions (joy, hope, pride, curiosity) and low levels of negative emotions (frustration, shame, hopeless, boredom, contempt, confusion, and sadness). The positive emotion intensities were higher in this profile compared to those of the other profiles. Thus, we labeled this profile as the positive emotion profile. The final profile (n = 39) was characterized by medium levels for all emotions. When compared to the other profiles, the most distinct feature of this group was their increased levels of negative emotion intensities for all negative emotions. Therefore, we referred to this group as the negative emotion profile. A multivariate analyses of variance (MANOVA) revealed that the emotion profiles significantly differed in regard to their mean emotion intensities [Wilks’s λ (28, 320) = 0.100, p < 0.001, η² = 0.68].

FIGURE 2

Figure 2. Comparison of mean emotion intensities between profiles.

TABLE 3

Table 3. Means and standard deviations for emotion items, emotion regulation, and learning measures by profile solutions.

Linking Emotion Profiles and Learning Outcomes

Differences in learning outcomes between profiles were analyzed using a latent growth linear mixed effect model. More specifically, we predicted learning outcomes with time (pre and post test) and profile membership as fixed factors and included a random intercept⁵ based on previous studies that showed the importance of individual differences in prior knowledge when learning with MetaTutor (Taub et al., 2014). The model explained significant proportion of variance in learning outcomes (R² = 68.03%; fixed effects: R² = 16.23%) and showed that learning outcomes significantly improved over time for all profiles [β = 0.75, SE = 0.06, t(175) = 12.36, p < 0.001, VIF = 1.00] and that membership in the negative profile was associated with significantly lower learning outcomes [β = −0.40, SE = 0.16; t(173) = −2.43, p < 0.05, VIF = 1.18; see Figure 3]⁶. Post hoc test using Tukey’s HSD (honestly significant difference) showed that significant differences in learning outcomes were only found between the negative and neutral profile (z = −2.432; p < 0.05).

FIGURE 3

Figure 3. Pre and post test scores by emotion profile.

Linking Emotion Profiles and Emotion Regulation

Two separate ANOVAs comparing expressive suppression and cognitive reappraisal between the profiles were conducted to test if profiles differed in their self-reported habitual use of emotion regulation strategies. Results showed that there were no significant differences in expressive suppression [F(2,163) = 0.013; p = 0.99], but significant differences in cognitive reappraisal between profiles [F(2,163) = 4.185; p < 0.05]. Post hoc comparisons using Bonferroni correction revealed that students with a negative emotion profile had significantly lower cognitive reappraisal scores (M = 4.62, SD = 1.27) then those with a positive emotion profile (M = 5.30, SD = 1.07; p < 0.05).⁷

Variable-Centered Approach: Patterns of Co-occurring Emotions

Identifying Patterns of Co-occurring Emotions

To identify patterns of co-occurring emotions, correlation matrices for the 14 emotions investigated in this study were computed separately for each point of time (see procedure) and aggregated over all points of time. Then, spectral co-clustering, a clustering technique that groups data by rows and columns simultaneously (e.g., Kluger et al., 2003), was applied to these matrices to obtain the variable-centered patterns of related emotions for each point of time and aggregated over all EV administrations. This procedure was carried out for cluster solutions ranging from three to six clusters. A four-cluster solution was the only one that displayed great stability over all time points and aggregated over all measures (the only exception is that contempt moved to the boredom cluster during the last measurement). This solution included a positive and a negative emotions pattern, as well as neutral and boredom as singular-emotion clusters (see Table 4). Cronbach’s Alpha was calculated for the negative and positive emotions pattern separately for each time point to test if the identified cluster represented an internally consistent linear structure sufficiently well. Results showed that both the negative pattern (alpha ranging from 0.74 to 0.81) and the positive pattern (alpha ranging from 0.72 to 0.85) met this criterion.⁸ We obtained participants’ individual scores for each pattern and the maintained variance of each pattern through principal component analyses with one component. The maintained variance from the original Likert-scale items for each non-singular emotion pattern was sufficient in this solution (35.45% for the negative emotions pattern for EV2 and 68.40% for the positive emotions pattern for EV2, see Table 4). Loadings for all emotions were positive for each pattern (i.e., increases in emotion intensity was associated with an increase in pattern score).

TABLE 4

Table 4. Maintained variance and loadings for emotion patterns.

Exploring Differences in Variable-Centered Emotion Patterns Scores Between Emotion Profiles

Differences in emotion variable-centered cluster scores between profiles over time were analyzed using latent growth linear mixed effect models. More specifically, we predicted variable-centered emotion pattern scores with time (six administrations of the EV), profile membership and their interaction as fixed factors and included a random intercept for the negative, positive and boredom emotion patterns.⁹ The model for the neutral emotion pattern did not include the interaction term of time and profile membership as the addition of this factor did not improve the model significantly. Results showed significant differences in emotion pattern scores on average for all emotion clusters (all p < 0.001). Furthermore, the negative, positive, and boredom pattern scores showed significant linear growth for all participants (all p < 0.001). For negative emotion pattern scores (R² = 62.99%, fixed effects: R² = 40.54%) we found significantly different linear trajectories between the negative profile and the other profiles [compared to neutral profile: β = 0.22, SE = 0.05, t(877) = 4.57, p < 0.001, VIF = 4.44; compared to positive profile: β = 0.21, SE = 0.05; t(877) = 4.16, p < 0.001, VIF = 4.11; see Figure 4]. Linear growth in positive emotion pattern scores (R² = 66.10%, fixed effects: R² = 40.44%) were significantly different between the positive and other profiles [compared to neutral profile: β = 0.08, SE = 0.04, t(877) = 1.99, p < 0.05, VIF = 3.17; compared to negative profile: β = 0.17, SE = 0.05, t(877) = 3.43, p < 0.001, VIF = 2.59]. Boredom pattern scores (R² = 56.99%, fixed effects: R² = 25.58%) illustrated significantly different linear trajectories between the positive and the neutral profile [β = 0.14, SE = 0.05, t(877) = 3.14, p < 0.010, VIF = 3.28].

FIGURE 4

Figure 4. Emotion pattern scores by emotion profile over the six measurement points.

Linking of Co-occurring Emotions and Learning Outcomes

To asses if variable-centered emotion patterns can predict learning gains, separate linear regression models predicting post test score with pretest and variable-centered emotion pattern scores for each point of time were calculated. Results showed that pretest score was a significant predictor of post score in all regressions (β ranging from 0.58 to 0.62; p < 0.01). The explanatory value of variable-centered emotion pattern scores beyond the effect of pretest score throughout the different points of time varied. The positive emotions pattern was the only significant predictor besides pretest score for the first administration of the EV [before learning sub goals were set; F(5,170) = 26.03, R² = 0.42; β = 0.15; p < 0.05] and a marginally significant predictor for the second administration [after learning sub goals were set; F(5,170) = 25.30, R² = 0.41; β = 0.14; p = 0.057]. Negative emotions pattern scores significantly predicted post test score for the fourth [second EV during the learning activity; F(5,170) = 24.22, R² = 0.40; β = −0.13; p < 0.05] and sixth administrations of the EV [directly before the post test; F(5,170) = 25.08, R² = 0.41; β = −0.17; p < 0.05] and were a marginally significant predictor for the third [first EV during the actual learning activity; F(5,170) = 24.05, R² = 0.40; β = −0.11; p = 0.086] and fifth EVs [last EV during the learning activity; F(5,170) = 23.01, R² = 0.39; β = −0.11; p = 0.082]. The other patterns showed no significant relation to post test score at any time point.

Discussion

This study used a person-centered approach to identify emotion profiles and a variable-centered approach to identify variable-centered emotion patterns throughout different phases of a learning session with MetaTutor. We further explored how the emotion profiles and variable-centered patterns identified through these approaches relate to learning outcomes (i.e., through a latent growth linear mixed effect model), and to self-reported habitual emotion regulations strategies.

With the person-centered approach we identified three distinct emotion profiles that reflected different emotional experiences during learning with MetaTutor. In line with our hypotheses and previous research, these profiles included a positive, negative, and neutral (referred to as low intensity in other studies; Robinson et al., 2017) emotion profile. However, it is important to note that the negative profile was not characterized by high levels of negative emotion intensities. It rather represented a group of students that had higher levels of negative emotions than the students belonging to the other profiles. An exception to this pattern was boredom, as the neutral profile showed comparable levels of boredom. This is in line with findings of previous studies emphasizing the distinct role of boredom during learning (Goetz et al., 2014). These findings were further supported through the variable-centered emotion patterns we identified in subsequent steps. Across six points of time throughout the learning session negative and positive emotions remained separate variable-centered patterns from boredom and neutral. This indicates that the separating features of our emotion profiles are related to a stable cluster structure of emotions. Moreover, our results indicated that the most profound difference in emotional experience between emotion profiles were found for the negative emotions (η² = 0.48 for the negative emotion cluster scores as compared to η² = 0.09 for other emotion cluster scores). In our profile solutions negative emotions were associated with one another regardless of their level of arousal. Interestingly, surprise was associated with the negative profile and negative emotions cluster. This finding corresponds with findings of a previous study that found a significant negative relation between surprise and the accuracy of metacognitive judgments indicating a potential negative impact on learning (Taub et al., 2019). However, the lack of differentiation of levels of arousal is likely caused by the imbalanced nature of arousal and valence in emotions measured in the present study (Robinson et al., 2017). Particularly, positive deactivating emotions were underrepresented in the EV. Nonetheless, across two different approaches we identified a theoretically supported and meaningful structure of emotions that centered around three levels of valence—i.e., positive, neutral, and negative.

The most striking feature across all profile solutions was the stability of the negative profile. More specifically, 26 of the 39 (67%) students in the negative profile were always assigned to the same profile regardless of the number of other profiles.¹⁰ This indicates that the group of students with higher levels of negative emotions is most distinct from all other students (in regard to emotional experience). More importantly, comparisons of the learning outcomes for the profiles revealed that the negative profile performed significantly worse than at least one other profile at post-test in most profile solutions. In the three-profile solution presented in this paper, the negative profile was significantly outperformed by the neutral profile. This finding is well in line with previous studies using person-centered approaches, as multiple studies found that students with negative emotion profiles tend to learn less than those with neutral or positive profiles (Ganotice et al., 2016; Jarrell et al., 2017; Robinson et al., 2017; see Table 2). As opposed to variable-centered approaches that showed positive and negative effects of negative and positive emotions depending on the circumstances, person-centered approaches consistently found detrimental effects of negative emotions for learning. While under certain circumstances single negative (resolved) emotions can potentially benefit learning strategies and outcomes (e.g., D’Mello and Graesser, 2014; Taub et al., 2019), our data provided no support for beneficial effects of experiencing multiple negative emotions (e.g., students that belong to a negative emotion profile). It is important to note that while mixed effects of positive and negative emotions depending on the circumstances have been found in multiple studies, most studies indicate that positive emotions are typically beneficial and negative emotions are detrimental for learning (Boekaerts and Pekrun, 2015). Our results supported this general trend for negative emotions.

In addition to the question of which profiles do significantly differ in learning, we also investigated if and how variable-centered emotion patterns would predict learning. We found that positive emotions before the actual learning activity (EVs 1 and 2, see section Emotion Items) can predict learning outcomes beyond the explanatory effect of prior knowledge. During self-regulated learning with MetaTutor only negative emotions were significant predictors of learning, but not consistently (significant for EV3 and EV6, only marginally significant for EV4 and EV5). These findings indicate that predictive value of variable-centered emotion patterns for learning fluctuates over time and that negative emotions seem to play a predominant role during the learning activity. Furthermore, these finding reflect central approaches related to learning in digital learning environments – products and processes (Garcia-Martin and Garcia-Sanchez, 2018). More specifically, the profile analysis conducted in this study is primarily product focused as we first investigated differences in learning outcomes (i.e., product data) between emotion profiles. With subsequent analyses, we investigated the process nature of emotions by assessing how emotions form patterns over time and how linear developments in these patterns are related to learning.

We faced several challenges and identified limitations when applying the two clustering approaches to the present data. Our sampling approach was defined relative to the start and end of the session. In particular, we selected the first two EVs and the last two in the learning session. Of these questionnaires, only the first in the learning phase (EV3) and the very last before the posttest (EV6) were administered identically for all participants. The EVs in between these were identical relative to the start and end of the learning session, but slightly different in regard to learning time depending on the total number of EVs the participant completed (e.g., for participants with six EVs all questionnaires were in an actual sequence, while for participants with eight EVs the new sequence included the first four EVs and the last two EVs, leaving two EVs out and creating a spline which might not completely reflect the initial temporal trajectory). However, both profile analyses across all time points and the emotion clusters revealed that the selected clusters represented a stable, comparable selection of measures over time.

As a potential explanation for differences between emotion profiles we compared them in regard to emotion regulation and found significant differences in cognitive reappraisal, but not for expressive suppression between profiles. More specifically, the negative profile reported significantly lower habitual use of cognitive reappraisal than the positive profile, but not compared to the neutral profile. To back up these findings we compared the profiles in regard to variable-centered emotion pattern scores and their linear temporal trajectories. We found that emotion profiles did not only differ in averaged emotion pattern scores for all identified emotion patterns but also exhibited significantly different linear growth for negative emotions, positive emotions and boredom (see Figure 4). The most distinct differences lied in the negative emotion pattern as the negative profile displayed a linear increase in negative emotion pattern scores while the scores decreased/stagnated in the other profiles. This illustrates that the negative profile not only starts with higher values of negative emotions, but that this difference got larger over time. Taken together with our finding that the negative emotions cluster negatively predicted learning throughout the learning phase, this indicates that the issues of the negative emotion profile seem to arise over time and are linked to emotion regulation.

A potential explanation for the suboptimal performance of the negative emotion profile is the potential load on working memory imposed by negative emotions and emotion regulation (Curci et al., 2013). While positive emotions cannot enhance working memory beyond its natural capacity, multiple negative emotions may block valuable resources that are particularly required for mastering complex topics and completing challenging learning tasks. This phenomenon might be even more important in digital learning environments as they impose significant challenges to learners (e.g., for navigation through non-linear hyperlinked environments, coordinating multiple goals, integrate agent feedback, use sophisticated learning strategies; Opfermann et al., 2013). Future studies aiming to explain why negative emotions pose a detrimental effect on learning are needed, including cognitive load and its relation to working memory (Seufert, 2018; Anmarkrud et al., 2019).

Another limitation of the present study (and person-centered approaches in general) is the decontextualized nature of emotion measures used. Theories on affective dynamics stretch the importance of specific events or impasses that elicit emotions, however, the events preceding the measurement of emotions have not been considered yet. Specifically, given our data we cannot disentangle whether students learned less because they experienced negative emotions or if they experienced negative emotions because they were having difficulties during the learning process. Identifying if the elevated levels of negative emotions in negative profiles is related to characteristics of the learning task or the learning environment is crucial for both the understanding of the profiles and the development of adaptive systems that can support students and circumvent negative effects of negative emotions on learning though scaffolds. For instance, in our study we cannot rule out that the increase in negative emotion, especially in the negative emotions profile, was related participants being prompted to fill out self-reports to indicate their emotions repeatedly during the learning activity. Likewise, the precedents of emotional reactions during learning should be incorporated in future studies (e.g., by assessing which emotions specific prompts of pedagogical agents elicit). Taub et al. (2019) have shown that facially expressed emotions are associated with the accuracy of learning strategies. Identifying arising negative emotions and the learning processes they directly affect can bridge the gap between emotions and (meta)-cognitive processes. This goes hand in hand with another shortcoming of this line of inquiry – the sole reliance on self-reports to measure emotions. Models and research on emotions clearly state that emotions are multi-faceted processes and limiting our scope to the appraisal component (Scherer and Moors, 2019) is a significant limitation. Building multi-channel, multi-modal emotion profiles through the use of additional data channels can benefit person-centered research by refining profiles and by providing additional explanations how the profiles develop over time (e.g., through peaks in EDA). Lastly, personal predispositions (e.g., personality – narcissism as a predisposition for negative emotionality) is a general cause for differences in emotional experience and emotion regulation, and its effect on learning strategies could be very beneficial to deepen the understanding of emotions in self-regulated learning processes.

Conclusion

In conclusion, the results of our study highlight the importance of negative emotions during self-regulated learning with digital learning environments during complex learning. The present study adds to research in multiple ways. Methodologically, we have showcased how a person-centered and a novel variable-centered approach complement each other. Particularly identifying variable-centered emotion patterns in addition to emotion profiles enabled us to analyze temporal dynamics of multiple emotions simultaneously. A negative relation between negative emotions and learning outcomes was found with both approaches. This underlines the robustness of this finding and further shows that person-centered and variable-centered approaches can supplement each other. Moreover, clustering approaches offer the possibility to further connect findings from studies using different measures more easily (e.g., achievement emotions vs. learning-centered emotions). Through the combination of person-centered and variable-centered approaches, we have found that both the students with the highest levels of negative emotions overall and higher levels of negative emotions across all students showed a significant negative relation to learning. Furthermore, we have found that these detrimental effects are linked to lower (self-reported) emotion regulation. This indicates the need to identify when elevated levels of negative emotions arise, particularly for students who experience a multitude of negative emotions, for practitioners and researchers to intervene in a timely fashion before the detrimental effects of negative emotions settle in. Specifically, fostering students’ emotion regulation as part of self-regulated learning activities with digital learning environments is a promising prospect to improve students’ emotional experience and learning subsequently. Therefore, the design, development, and implementation of digital learning environments as well as educational interventions should incorporate emotions and emotion regulation as parts of (self-regulated) learning activities to maximize positive effects on students’ learning.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

This study was approved the by North Carolina State University’s IRB. All subjects gave written consent in accordance with the Declaration of Helsinki.

Author Contributions

All authors contributed to the conception of the work and revised the final manuscript. RA and MT designed and conducted the study. FW conducted the statistical analyses. FW and MT wrote the first draft of the manuscript. RA and SN provided several rounds of edits on the manuscript.

Funding

This study was supported by funding from the National Science Foundation (DRL#1660878, DRL#1661202, DUE#1761178, and DRL#1916417) and the Social Sciences and Humanities Research Council of Canada (SSHRC 895-2011-1006). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or Social Sciences and Humanities Research Council of Canada. FW was funded by the LEAD Graduate School and Research Network (GSC1028), a project of the Excellence Initiative of the German federal and state governments. We acknowledge the support by the Deutsche Forschungsgemeinschaft and Open Access Publishing Fund of the University of Tübingen.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors would like to thank the members of the SMART Lab at UCF for their assistance and contributions.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.02678/full#supplementary-material

Footnotes

1. ^ The dynamics of affective states model and related research often refer to cognitive-affective states instead of emotions. For consistency, readability, and because the cognitive component of these states resembles the appraisal component of emotion theories (e.g., Moors et al., 2013) we will refer to them as emotions. However, we acknowledge arguments that these terms might not be interchangeable in all contexts.
2. ^ Analyses scripts and data are available upon request. For analyses that required (pseudo) randomization, seeds used to obtain the results reported in this paper were documented to guarantee replicability.
3. ^ All subsequent analyses were also conducted for the six- and seven profile-solution. The pattern of results remained similar. These results were not reported in this study due to space constraints.
4. ^ Labels for the profiles were chosen based on the most dominant feature overall and in comparison to the other profiles.
5. ^ The step-wise model selection procedures were not reported due to space constraints. They can be found in the Supplementary Material.
6. ^ This pattern of results was consistent for all profile solutions of the two-step approach but only for the three-profile solution in the initial clustering approach.
7. ^ Further analyses revealed that this pattern of results stayed identical across all profile solutions (four- and five-profile solution from the k-means clustering as well as three- to five-profile solutions from the initial hierarchical clustering analyses).
8. ^ No item had to be rescaled for these analyses, showing that all measures in the pattern correlated positively with the pattern score assigned to each participant.
9. ^ Random slopes were initially considered but lead to potentially overfitted models (singular fit) and were therefore not considered in final analyses. The model selection summary can be found in the Supplementary Material.
10. ^ This pattern was even stronger in the hierarchical cluster analyses as over 90% of students in that profile were consistently assigned to the same profile regardless of the number of other profiles.

References

Anmarkrud, Ø, Andresen, A., and Bråten, I. (2019). Cognitive load and working memory in multimedia learning: conceptual and measurement issues. Educ. Psychol. 54, 1–23.

Google Scholar

Arguel, A., Lockyer, L., Kennedy, G., Lodge, J. M., and Pachman, M. (2019). Seeking optimal confusion: a review on epistemic emotion management in interactive digital learning environments. Interact. Learn. Environ. 27, 200–210. doi: 10.1080/10494820.2018.1457544

CrossRef Full Text | Google Scholar

Asendorpf, J. B., Borkenau, P., Ostendorf, F., and Van Aken, M. A. G. (2001). Carving personality description at its joints: confirmation of three replicable personality prototypes for both children and adults. Eur. J. Pers. 15, 169–198. doi: 10.1002/per.408.abs

CrossRef Full Text | Google Scholar

Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-Behnagh, R., Bouchet, F., et al. (2013). “Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems,” in International Handbook of Metacognition and Learning Technologies, eds R. Azevedo, and V. Aleven, (New York, NY: Springer), 427–449. doi: 10.1007/978-1-4419-5546-3_28

CrossRef Full Text | Google Scholar

Azevedo, R., Martin, S. A., Taub, M., Mudrick, N. V., Millar, G. C., and Grafsgaard, J. F. (2016). “Are pedagogical agents’ external regulation effective in fostering learning with intelligent tutoring systems?,” in Intelligent Tutoring Systems. ITS 2016, eds A. Micarelli, J. Stamper, and K. Panourgia, (Zagreb, Croatia: Springer).

Google Scholar

Azevedo, R., Mudrick, N. V., Taub, M., and Bradbury, A. (2019). “Self-regulation in computer-assisted learning systems,” in Handbook of Cognition and Education, eds J. Dunlosky, and K. Rawson, (Cambridge, MA: Cambridge University Press).

Google Scholar

Azevedo, R., Mudrick, N. V., Taub, M., and Wortha, F. (2017). “Coupling between metacognition and emotions during STEM learning with advanced learning technologies: A critical analysis, implications for future research, and design of learning systems,” in Self-Regulated Learning: Conceptualization, Contribution, and Empirically Based Models for Teaching and Learning, eds T. Michalsky, and C. Schechter, (New York, NY: Teachers College Press)

Google Scholar

Azevedo, R., Taub, M., and Mudrick, N. V. (2018). “Using multi-channel trace data to infer and foster self-regulated learning between humans and advanced learning technologies,” in Handbook of Self-Regulation of Learning and Performance, eds D. Schunk, and J. A. Greene, (New York, NY: Routledge).

Google Scholar

Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., and Graesser, A. C. (2010). Better to be frustrated than bored: the incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. Int. J. Hum. Comput. Stud. 68, 223–241. doi: 10.1016/j.ijhcs.2009.12.003

CrossRef Full Text | Google Scholar

Barker, E. T., Howard, A. L., Galambos, N. L., and Wrosch, C. (2016). Tracking affect and academic success across university: happy students benefit from bouts of negative mood. Dev. Psychol. 52, 2022–2030. doi: 10.1037/dev0000231

PubMed Abstract | CrossRef Full Text | Google Scholar

Bates, D., Mächler, M., Bolker, B., and Walker, S. (2014). Fitting linear mixed-effects models using lme4. J. Stat. Soft. 67, 1–48.

Google Scholar

Baumeister, R. F., Alquist, J. L., and Vohs, K. D. (2015). Illusions of learning: irrelevant emotions inflate judgments of learning. J. Behav. Decis. Mak. 28, 149–158. doi: 10.1002/bdm.1836

CrossRef Full Text | Google Scholar

Behrendt, S. (2014). Lm.Beta: Add Standardized Regression Coefficients To Lm-Objects. Available at: https://CRAN.R-project.org/package=lm.beta (accessed November 20, 2019).

Google Scholar

Ben-Eliyahu, A., and Linnenbrink-Garcia, L. (2013). Extending self-regulated learning to include self-regulated emotion strategies. Motiv. Emot. 37, 558–573. doi: 10.1007/s11031-012-9332-3

CrossRef Full Text | Google Scholar

Boekaerts, M., and Pekrun, R. (2015). “Emotions and emotion regulation in academic settings,” in Handbook of Educational Psychology, eds L. Corno, and E. M. Anderman, (New York, NY: Routledge)Google Scholar

Breckenridge, J. N. (2000). Validating cluster analysis: consistent replication and symmetry. Multivariate Behav. Res. 35, 261–285. doi: 10.1207/S15327906MBR3502_5

PubMed Abstract | CrossRef Full Text | Google Scholar

Charrad, M., Ghazzali, N., Boiteau, V., Niknafs, A., and Charrad, M. M. (2014). Package ‘nbclust’. J. Stat. Softw. 61, 1–36.

Google Scholar

R Core Team (2019). R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing

Google Scholar

Craig, S., Graesser, A., Sullins, J., and Gholson, B. (2004). Affect and learning: an exploratory look into the role of affect in learning with Autotutor. J. Edu. Media 29, 241–250. doi: 10.1080/1358165042000283101

CrossRef Full Text | Google Scholar

Curci, A., Lanciano, T., Soleti, E., and Rimé, B. (2013). Negative emotional experiences arouse rumination and affect working memory capacity. Emotion 13, 867–880. doi: 10.1037/a0032492

PubMed Abstract | CrossRef Full Text | Google Scholar

D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. J. Educ. Psychol. 105:1082. doi: 10.1037/a0032674

CrossRef Full Text | Google Scholar

D’Mello, S. K. (2017). “Emotional learning analytics,” in Handbook of Learning Analytics & Educational Data Mining, 1st Edn, eds C. Lang, G. Siemens, W. Alyssa, and D. Gašević (Edmonton, AB: Society for Learning Analytics Research).

Google Scholar

D’Mello, S., and Graesser, A. (2012). Dynamics of affective states during complex learning. Learn. Instr. 22, 145–157. doi: 10.1016/j.learninstruc.2011.10.001

CrossRef Full Text | Google Scholar

D’Mello, S., and Graesser, A. (2014). Confusion and its dynamics during device comprehension with breakdown scenarios. Acta Psychol. 151, 106–116. doi: 10.1016/j.actpsy.2014.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

D’Mello, S., Lehman, B., Pekrun, R., and Graesser, A. (2014). Confusion can be beneficial for learning. Learn. Instr. 29, 153–170. doi: 10.1016/j.learninstruc.2012.05.003

CrossRef Full Text | Google Scholar

Ekman, P. (1992). An argument for basic emotions. Cogn. Emot. 6, 169–200. doi: 10.1016/j.concog.2017.10.008

PubMed Abstract | CrossRef Full Text | Google Scholar

Ekman, P., and Friesen, W. V. (1971). Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17, 124–129. doi: 10.1037/h0030377

CrossRef Full Text | Google Scholar

Eshghi, A., Haughton, D., Legrand, P., Skaletsky, M., and Woolford, S. (2011). Identifying groups: a comparison of methodologies. J. Data Sci. 9, 271–292.

Google Scholar

Fernando, J. W., Kashima, Y., and Laham, S. M. (2014). Multiple emotions: a person-centered approach to the relationship between intergroup emotion and action orientation. Emotion 14:722. doi: 10.1037/a0036103

PubMed Abstract | CrossRef Full Text | Google Scholar

Fortunato, V. J., and Goldblatt, A. M. (2006). An examination of goal orientation profiles using cluster analysis and their relationships with dispositional characteristics and motivational response patterns. J. Appl. Soc. Psychol. 36, 2150–2183. doi: 10.1111/j.0021-9029.2006.00099.x

CrossRef Full Text | Google Scholar

Ganotice, F. A. Jr., Datu, J. A. D., and King, R. B. (2016). Which emotional profiles exhibit the best learning outcomes? a person-centered analysis of students’ academic emotions. Sch. Psychol. Int. 37, 498–518. doi: 10.1177/0143034316660147

CrossRef Full Text | Google Scholar

Garcia-Martin, J., and Garcia-Sanchez, J. N. (2018). The instructional effectiveness of two virtual approaches: processes and product. Rev. Psicodidáctica 23, 117–127. doi: 10.1016/j.psicoe.2018.02.003

CrossRef Full Text | Google Scholar

Gegenfurtner, A., Fryer, L., Järvelä, S., Harackiewicz, J., and Narciss, S. (2019). Affective learning in digital education. Front. Educ.

Google Scholar

George, D., and Mallery, P. (1999). SPSS^® for Windows^® Step By Step: a Simple Guide and Reference. Needham Heights, MA: Allyn & Bacon.

Google Scholar

Goetz, T., Frenzel, A. C., Hall, N. C., Nett, U. E., Pekrun, R., and Lipnevich, A. A. (2014). Types of boredom: an experience sampling approach. Motiv. Emot. 38, 401–419. doi: 10.1007/s11031-013-9385-y

CrossRef Full Text | Google Scholar

Graesser, A. C., Rus, V., D’Mello, S., and Jackson, G. (2008). “AutoTutor: learning through natural language dialogue that adapts to the cognitive and affective states of the learner,” in Current Perspectives on Cognition, Learning and Instruction: Recent Innovations In Educational Technology That Facilitate Student Learning, eds D. H. Robinson, and G. Schraw, (Charlotte, NC: Information Age Publishing), 95–125.

Google Scholar

Grafsgaard, J., Wiggins, J., Vail, A., Boyer, K., Wiebe, K., and Lester, J. (2014). “The additive value of multimodal features for predicting engagement, frustration, and learning during tutoring,” in Proceedings of The Sixteenth Acm International Conference on Multimodal Interaction, eds A. A. Salah, J. Cohn, and B. Schuller, (New York, NY: ACM).

Google Scholar

Gross, J. J. (2015). Emotion regulation: current status and future prospects. Psychol. Inq. 26, 1–26. doi: 10.1080/1047840x.2014.940781

CrossRef Full Text | Google Scholar

Gross, J. J., and John, O. P. (2003). Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J. Pers. Soc. Psychol. 85, 348–362. doi: 10.1037/0022-3514.85.2.348

PubMed Abstract | CrossRef Full Text | Google Scholar

Grubbs, F. E. (1969). Procedures for detecting outlying observations in samples. Technometrics 11, 1–21. doi: 10.1080/00401706.1969.10490657

CrossRef Full Text | Google Scholar

Harley, J. M., Pekrun, R., Taxer, J. L., and Gross, J. J. (2019). Emotion regulation in achievement situations: an integrated model. Edu. Psychol. 54, 106–126. doi: 10.1080/00461520.2019.1587297

CrossRef Full Text | Google Scholar

Harley, J. M., Taub, M., Azevedo, R., and Bouchet, F. (2017). Let’s set up some subgoals: understanding human-pedagogical agent collaborations and their implications for learning and prompt and feedback compliance. IEEE Trans. Learn. Technol. 11, 54–66. doi: 10.1109/tlt.2017.2756629

CrossRef Full Text | Google Scholar

Hothorn, T., Bretz, F., and Westfall, P. (2008). Simultaneous inference in general parametric models. Biometrical J. 50, 346–363. doi: 10.1002/bimj.200810425

CrossRef Full Text | Google Scholar

Jarrell, A., Harley, J. M., Lajoie, S., and Naismith, L. (2017). Success, failure and emotions: examining the relationship between performance feedback and emotions in diagnostic reasoning. Edu. Technol. Res. Dev. 65, 1263–1284. doi: 10.1007/s11423-017-9521-6

CrossRef Full Text | Google Scholar

Jarrell, A., Harley, J. M., and Lajoie, S. P. (2016). The link between achievement emotions, appraisals, and task performance: pedagogical considerations for emotions in CBLEs. J. Comput. Edu. 3, 289–307. doi: 10.1007/s40692-016-0064-3

CrossRef Full Text | Google Scholar

Kluger, Y., Basri, R., Chang, J. T., and Gerstein, M. (2003). Spectral biclustering of microarray data: coclustering genes and conditions. Genome Res. 13, 703–716. doi: 10.1101/gr.648603

PubMed Abstract | CrossRef Full Text | Google Scholar

Komsta, L. (2011). Outliers: Tests for Outliers. Available at: https://CRAN.R-project.org/package=outliers (accessed November 20, 2019).

Google Scholar

Lazarus, R. S. (2006). Emotions and interpersonal relationships: toward a person-centered conceptualization of emotions and coping. J. Pers. 74, 9–46. doi: 10.1111/j.1467-6494.2005.00368.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Lewis, M., Haviland-Jones, J. M., and Barrett, L. F. (2008). Handbook of Emotions. New York, NY: The Guilford Press.

Google Scholar

Makowski, D. (2018). The psycho package: an efficient and publishing-oriented workflow for psychological science. J. Open Source Soft. 3:470. doi: 10.21105/joss.00470

CrossRef Full Text | Google Scholar

Moors, A., Ellsworth, P. C., Scherer, K. R., and Frijda, N. H. (2013). Appraisal theories of emotion: state of the art and future development. Emot. Rev. 5, 119–124. doi: 10.1177/1754073912468165

CrossRef Full Text | Google Scholar

Muis, K. R., Pekrun, R., Sinatra, G. M., Azevedo, R., Trevors, G., Meier, E., et al. (2015). The curious case of climate change: testing a theoretical model of epistemic beliefs, epistemic emotions, and complex learning. Learn. Instr. 39, 168–183. doi: 10.1016/j.learninstruc.2015.06.003

CrossRef Full Text | Google Scholar

Opfermann, M., Scheiter, K., Gerjets, P., and Schmeck, A. (2013). “Hypermedia and self-regulation: an Interplay in both directions,” in International Handbook Of Metacognition and Learning Technologies, eds R. Azevedo, and V. Aleven, (New York, NY: Springer New York).

Google Scholar

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830.

Google Scholar

Pekrun, R. (2006). The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ. Psychol. Rev. 18, 315–341. doi: 10.1007/s10648-006-9029-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Pekrun, R., Goetz, T., Titz, W., and Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: a program of qualitative and quantitative research. Educ. Psychol. 37, 91–105. doi: 10.1207/s15326985ep3702_4

CrossRef Full Text | Google Scholar

Pekrun, R., Lichtenfeld, S., Marsh, H. W., Murayama, K., and Goetz, T. (2017a). Achievement emotions and academic performance: longitudinal models of reciprocal effects. Child Dev. 88, 1653–1670. doi: 10.1111/cdev.12704

PubMed Abstract | CrossRef Full Text | Google Scholar

Pekrun, R., Vogl, E., Muis, K. R., and Sinatra, G. M. (2017b). Measuring emotions during epistemic activities: the epistemically-related emotion scales. Cogn. Emot. 31, 1268–1276. doi: 10.1080/02699931.2016.1204989

PubMed Abstract | CrossRef Full Text | Google Scholar

Pekrun, R., and Linnenbrink-Garcia, L. (2012). “Academic emotions and student engagement,” in Handbook of Research on Student Engagement, eds S. L. Christenson, A. L. Reschly, and C. Wylie, (Boston, MA: Springer).

Google Scholar

Pekrun, R., and Linnenbrink-Garcia, L. (2014). International Handbook of Emotions in Education. New York, NY: Routledge.

Google Scholar

Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., and Um, E. (2014). Emotional design in multimedia learning: effects of shape and color on affect and learning. Learn. Instr. 29, 128–140. doi: 10.1016/j.learninstruc.2013.02.006

CrossRef Full Text | Google Scholar

Revelle, W. R. (2017). psych: Procedures for Personality and Psychological Research (Version 1.8.12). Available at: https://CRAN.R-project.org/package=psych (accessed November 20, 2019).

Google Scholar

Riemer, V., and Schrader, C. (2019). Mental model development in multimedia learning: interrelated effects of emotions and self-monitoring. Front. Psychol. 10:899. doi: 10.3389/fpsyg.2019.00899

PubMed Abstract | CrossRef Full Text | Google Scholar

Robinson, K. A., Ranellucci, J., Lee, Y. K., Wormington, S. V., Roseth, C. J., and Linnenbrink-Garcia, L. (2017). Affective profiles and academic success in a college science course. Contemp. Educ. Psychol. 51, 209–221. doi: 10.1016/j.cedpsych.2017.08.004

CrossRef Full Text | Google Scholar

Sabourin, J. L., and Lester, J. C. (2014). Affect and engagement in game-based learning environments. IEEE Trans. Affect. Comput. 5, 45–56.

Google Scholar

Scherer, K., and Moors, A. (2019). The emotion process: event appraisal and component differentiation. Annu. Rev. Psychol. 70, 719–745. doi: 10.1146/annurev-psych-122216-011854

PubMed Abstract | CrossRef Full Text | Google Scholar

Seufert, T. (2018). The interplay between self-regulation in learning and cognitive load. Educ. Res. Rev. 24, 116–129. doi: 10.1016/j.edurev.2018.03.004

CrossRef Full Text | Google Scholar

Sinclair, J., Jang, E. E., Azevedo, R., Lau, C., Taub, M., and Mudrick, N. (2018). “Changes in emotion and their relationship with learning gains in the context of MetaTutor,” in Lecture notes in computer science: Intelligent tutoring systems 2018, eds R. Nkambou, R. Azevedo, and J. Vassileva, (Cham: Springer).

Google Scholar

Spann, C. A., Shute, V. J., Rahimi, S., and D’Mello, S. K. (2019). The productive role of cognitive reappraisal in regulating affect during game-based learning. Comput. Hum. Behav. 100, 358–369. doi: 10.1016/j.chb.2019.03.002

CrossRef Full Text | Google Scholar

SPSS, I. (2012). SPSS Version 21.0. IBM SPSS. Chicago, IL.

Google Scholar

Taub, M., Azevedo, R., Bouchet, F., and Khosravifar, B. (2014). Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Comput. Hum. Behav. 39, 356–367. doi: 10.1016/j.chb.2014.07.018

CrossRef Full Text | Google Scholar

Taub, M., Azevedo, R., Bradbury, A. E., Millar, G. C., and Lester, J. (2018). Using sequence mining to reveal the efficiency in scientific reasoning during STEM learning with a game-based learning environment. Learn. Instr. 54, 93–103. doi: 10.1016/j.learninstruc.2017.08.005

CrossRef Full Text | Google Scholar

Taub, M., Azevedo, R., Rajendran, R., Cloude, E. B., Biswas, G., and Price, M. J. (2019). How are students’ emotions related to the accuracy of cognitive and metacognitive processes during learning with an intelligent tutoring system? Learn. Instr.

Google Scholar

Tyng, C. M., Amin, H. U., Saad, M. N. M., and Malik, A. S. (2017). The influences of emotion on learning and memory. Front. Psychol. 8:1454. doi: 10.3389/fpsyg.2017.01454

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Rossum, G., and Drake, F. L. (2011). Python Language Reference Manual. Bristol: Network Theory Ltd.

Google Scholar

Vansteenkiste, M., Sierens, E., Soenens, B., Luyckx, K., and Lens, W. (2009). Motivational profiles from a self-determination perspective: the quality of motivation matters. J. Educ. Psychol. 101, 671–688. doi: 10.1037/a0015083

CrossRef Full Text | Google Scholar

Venables, W., and Ripley, B. (2002). Statistics Complements to Modern Applied Statistics with S, 4th Edn, New York, NY: Springer.

Google Scholar

Zheng, L. (2016). The effectiveness of self-regulated learning scaffolds on academic performance in computer-based learning environments: a meta-analysis. Asia Pacific Educ. Rev. 17, 187–202. doi: 10.1007/s12564-016-9426-9

CrossRef Full Text | Google Scholar