Epistemic Emotions and Epistemic Cognition Predict Critical Thinking About Socio-Scientific Issues

When thinking critically about socio-scientific issues, individuals’ expectations about the nature of knowledge and knowing, as well as their emotions when these expectations are met or not, may play an important role in critical thinking. In this study, we examined the role of epistemic emotions in mediating the effects of epistemic cognition on critical thinking when contending with conflicting information about genetically modified foods. Two hundred four university students completed a prior knowledge test on genetically modified foods, and then reported their epistemic beliefs about genetically modified foods. Participants then read a text that presented advantages and disadvantages of genetically modified foods, and reported the epistemic emotions they experienced during reading of that text. Participants then composed an argumentative essay about genetically modified foods, which were coded for critical thinking. Results from path analysis revealed that a belief in complex knowledge predicted less surprise and confusion, but more enjoyment. For the source of knowledge, a belief in the active construction of knowledge predicted less surprise and enjoyment. For justification for knowing, a belief that knowledge should be critically evaluated positively predicted curiosity, and negatively predicted confusion and boredom. Moreover, beliefs that knowledge about genetically modified foods is complex and uncertain positively predicted critical thinking. Confusion and anxiety also positively predicted critical thinking, whereas frustration negatively predicted critical thinking. Lastly, confusion mediated relations between epistemic beliefs and critical thinking. Results suggest complex relations between epistemic cognition, epistemic emotions, and critical thinking that have implications for educational practice as well as for future research on epistemic cognition and epistemic emotions.

Introduction

The information landscape in the 21st century is one of contrast. On the one hand, the Internet and social media provide an unprecedented wealth of diverse and accessible information from around the world. On the other hand, the structure of social networks and algorithmic filtering (e.g., news feeds and recommendations) have considerably narrowed the breadth of content that individuals consume, making it increasingly difficult to escape echo chambers and challenge one’s views with new information. In this context, any topic is likely to become the object of controversy. Topics of personal and global relevance such as ways to combat climate change or the safety of infant vaccines appear to be politically controversial, dividing the public’s opinion on what is considered accurate information, and stifling political action. To make informed decisions individually and collectively, the challenge lies in overcoming personal biases, and weighing the pros and cons of conflicting perspectives to reconcile views (Noroozi et al., 2018). This is one aspect of the process known as critical thinking (Kuhn, 2018).

There is little debate over the idea that society benefits when individuals are able to think deeply and critically about important issues (e.g., Dewey, 1933; Halpern, 2014). Educating people to become critical thinkers is of vital importance for the well-being of future generations. Accordingly, the Organization for Economic Cooperation and Development (OECD; Tremblay et al., 2012) has made teaching critical thinking a priority for higher education. However, empirical research shows that teaching critical thinking skills is arduous and often unyielding (Abrami et al., 2008; Niu et al., 2013; Huber and Kuncel, 2015), with up to 45% of students completing post-secondary degrees lacking these essential skills (Arum and Roksa, 2011). In light of these observations, many have suggested that to improve critical thinking outcomes, empirical work is needed to achieve a greater understanding of the underlying cognitive, motivational, and affective mechanisms that enable critical thinking (Alexander, 2014; Greene and Yu, 2014; Bråten, 2016).

Socio-scientific topics are often characterized by the presence of opposing views that offer conflicting explanations to complex and multifaceted phenomena (Levinson, 2006). Deciding what to believe or what to do about these topics requires that individuals engage with the underlying issues of knowledge that characterizes these topics: What counts as knowledge? How certain are the facts? Who can be trusted to provide a clear perspective on the topic? In other words, thinking critically about socio-scientific topics requires thinking about the knowledge- and knowing-related aspects of these issues (Greene and Yu, 2014), a process termed epistemic cognition (Greene et al., 2016). However, when engaged with complex and conflicting issues, individuals’ expectations about the nature of knowledge and knowing may be challenged, and in turn elicit emotions such as surprise, curiosity, confusion, frustration, or anxiety (Muis et al., 2018).

Common understandings of critical thinking assume that emotions have no role to play in critical thinking, except perhaps to introduce unwarranted bias (Kahneman, 2011). However, knowing and feeling¹ are closely related, and emotions may play a significant role in helping individuals disentangle the two (Brun and Kuenzle, 2008). For example, Tiedens and Linton (2001) suggest that emotions can serve as information about the state of certainty. To illustrate, when presented with a knowledge claim, feelings of uncertainty may lead an individual to doubt the veracity of that claim. This uncertainty may then lead to a more thorough treatment of information and a greater attention to the quality of arguments over the source’s characteristics. Nonetheless, little is known about how cognitive and affective processes interact to predict critical thinking. As such, the aim of the current study is to shed light on the role that epistemic cognition and epistemic emotions play when thinking critically about socio-scientific issues. In the following sections, we define the concepts of critical thinking, epistemic cognition, and epistemic emotions, and review theoretical and empirical work that informed the hypotheses of the current study.

Thinking Critically About Controversial Topics

Critical thinking is regarded as one of the most important skills that individuals can develop and is a fundamental aim of education (Bailin and Siegel, 2003; Halpern, 2014). Though several definitions of critical thinking are offered in the literature (e.g., Kurfiss, 1988; Siegel, 1988; Facione, 1990; Scriven and Paul, 1996; Litman, 2008; Ennis, 2018), Ennis (2018) argued that they do not significantly differ from each other. Drawing from these definitions, we define critical thinking as purposeful, reasonable and reflective thinking that enables individuals to decide what to believe or what to do when faced with complex and conflicting issues (Facione, 1990; Ennis, 2018). Following Kuhn (2018), we further define critical thinking as incorporating two key dimensions: inquiry (input), and argument (output).

According to Kuhn (2018), these two key dimensions of critical thinking can be delineated as an input phase and an output phase. Inquiry, the input phase, captures what an individual does as they are faced with complex and conflicting issues. Critical thinking during this phase includes skills like identifying pertinent information, evaluating claims, identifying counter-arguments, and critically analyzing and synthesizing information. These processes are carried out for the ultimate purpose of bringing this newly synthesized information to bear on a claim, which leads to the second dimension of critical thinking: argument.

Argument refers to a product that is constructed in written or oral form by an individual, which consists of a claim and one or more supporting reasons or evidence that are connected to the claim with warrants (Toulmin, 2003). Argumentation refers to the dynamic process that captures what is done to create the argument (Kuhn et al., 2015). As such, the output phase refers to the actions or processes of reasoning systematically in support of an idea, action or theory. Argumentation can be captured via dialogic methods (Kuhn, 2018) or via argumentative essay writing (Noroozi et al., 2018; Latifi et al., 2019; Valero Haro et al., 2019). For example, high-quality argumentative essays encompass a clear claim supported by evidence and reason, followed by acknowledgments of counter-arguments against the original claim, and integration of the arguments and counter-arguments which eventually lead to the final conclusion (Noroozi et al., 2016). The goal is to provide strong evidence to support one argument over another by weakening the other position (Kuhn, 2018).

Recent research has shown that critical thinking skills differ across academic disciplines (Gordon, 2000) given that various disciplines have different argumentation structures, epistemologies, and rules and goals (Noroozi et al., 2016). For instance, in nursing, critical thinking is concerned with rigorous investigation and reflection on all aspects of a clinical situation to decide on an appropriate course of action (Simpson and Courtney, 2002). In engineering, critical thinking consists of considering assumptions in problem-solving, selecting appropriate methods for experiments, structuring open-ended design problems, and assessing social impacts (Claris and Riley, 2012). When it comes to taking a position on a socio-scientific issue such as genetically modified foods, the task of critical thinking rests on identifying opposing arguments, assumptions, and evidence, evaluating the credibility, reliability, and relevance of claims, producing valid explanations and arguments, and making decisions or drawing valid conclusions (Facione, 1990; Kuhn and Crowell, 2011; Noroozi et al., 2016; Latifi et al., 2019).

Bailin and Siegel (2003), as well as other philosophical theorists of critical thinking (e.g., Paul, 1990), emphasized the importance of generalizable abilities such as assessing reasons, evaluating claims, identifying underlying assumptions, and recognizing and applying valid forms of justification. They argue that what is “critical” about critical thinking is the use of a criterion—an epistemic criterion—for evaluating reasons and making sound judgments. The generalizable reasoning abilities described by Bailin and Siegel (2003) have long been studied by educational and developmental psychologists in the field of epistemic cognition (e.g., King and Kitchener, 2002; Chinn et al., 2011; Hofer and Bendixen, 2012; Greene et al., 2016). Epistemic cognition concerns individuals’ thoughts and beliefs about the nature of knowledge and the process of knowing (Hofer and Pintrich, 1997). From the perspective of educational development, Kuhn (1991, 1999) identified the development of epistemic cognition as perhaps the most central underpinning of critical thinking.

The Role of Epistemic Cognition in Critical Thinking

Epistemic Cognition

Epistemic cognition refers to how individuals vet, acquire, understand, justify, and use knowledge (Greene et al., 2016). Specifically, individuals engage in epistemic cognition when they activate personal beliefs about the nature of knowledge and knowing (i.e., epistemic beliefs), define epistemic aims and criteria for knowing, and use evaluation and justification strategies to address issues of knowledge and knowing (Chinn et al., 2011; Barzilai and Zohar, 2014; Muis et al., 2018). The vast majority of research on epistemic cognition has focused on epistemic beliefs, which refer to individuals’ personal beliefs about the nature of knowledge and the process of knowing (Hofer and Pintrich, 1997). Hofer and Pintrich (1997) proposed that epistemic beliefs comprise four dimensions: (1) the complexity of knowledge, ranging from the belief that knowledge consists of a simple accumulation of facts, to the belief that knowledge consists of a complex structure of interrelated propositions; (2) the uncertainty of knowledge, ranging from the belief that knowledge is certain and unchanging, to the belief that knowledge is tentative and evolving; (3) the sources of knowing, ranging from the view that knowledge resides in external authorities, to the view that individuals are knowers who actively construct knowledge; and (4) the justification for knowing, which addresses how individuals evaluate knowledge claims, from an unquestioning reliance on authorities, to the evaluation and integration of evidence and arguments from various sources.

Numerous empirical studies have shown that individuals who adopt more constructivist epistemic cognition (e.g., who believe that knowledge is complex, tentative, actively constructed, and justified via evaluation) use better learning strategies (Chiu et al., 2013; Muis et al., 2015), show better self-regulation during problem solving (Muis et al., 2015), and attain greater academic performance (Bråten et al., 2014) than those who adopt less constructivist epistemic cognition (i.e., who believe that knowledge is simple, certain, handed down from, and justified by authorities).

Relations Between Epistemic Cognition and Critical Thinking

Across multiple studies, more constructivist epistemic cognition has been positively associated with critical thinking. Specifically, constructivists are better at identifying the elements of discourse (i.e., assumptions, evidence, arguments; Mason and Boscolo, 2004) and understanding authors’ viewpoints (Barzilai and Eshet-Alkalai, 2015) when reading texts that comprise conflicting perspectives, compared to individuals with less constructivist epistemic cognition. Similarly, when contending with multiple sources of information, individuals who engage in more constructivist epistemic cognition performed better at evaluating the trustworthiness and credibility of information using the features of the sources, distinguishing between types of sources, making associations between a source and its content, using criteria to evaluate the trustworthiness of sources, and using source integration strategies than those with less constructivist views (Barzilai and Zohar, 2012; Bråten et al., 2014; Strømsø and Bråten, 2014; McGinnis, 2016).

More constructivist beliefs about the justification for knowing have been associated with the use of more competent criteria to evaluate the trustworthiness of sources (Strømsø et al., 2011). Moreover, learners with more constructivist epistemic cognition have been found to possess greater argumentative skills (Mason and Boscolo, 2004; Yang and Tsai, 2010; Noroozi, 2018). Constructivists are also better able to support their statements with acceptable, relevant, and multiple justifications (Mason and Scirica, 2006). In sum, individuals who engage in more constructivist epistemic cognition are more likely to possess the cognitive skills necessary to think critically. In support of this, Muis and Duffy (2013) found that graduate students who received an intervention designed to develop more constructivist epistemic beliefs over the course of a semester also showed more critical thinking when learning statistics.

Research has also shown that, compared to less constructivist epistemic cognition, more constructivist epistemic cognition has been related to the will to take on multiple perspectives, reconsider one’s own thinking when drawing conclusions about controversial issues (Schommer-Aikins and Hutter, 2002), engage in effortful thinking (Hyytinen et al., 2014), and display skepticism toward unreliable sources (McGinnis, 2016). Though motivational and affective dispositions have theoretically been proposed to support critical thinking within the epistemic cognition literature (Chinn et al., 2011; Muis et al., 2015, 2018), little research has been conducted to understand how epistemic cognition relates to the affective states that dispose learners to think critically.

Epistemic Emotions and Critical Thinking

There is increasing evidence for the important role of emotions for learning processes and outcomes. Empirical research has related emotions to academic motivation, knowledge building and revision, as well as academic performance (Pekrun and Linnenbrink-Garcia, 2014). Broadly, emotions are defined by interrelated psychological processes that include affective (e.g., feeling nervous), cognitive (e.g., ruminating thoughts), motivational (e.g., a desire to escape), expressive (e.g., displaying a frown), and physiological (e.g., increased heart rate) components (Ellsworth, 2013; Shuman and Scherer, 2014). Emotions can generally be classified in terms of valence, where pleasant emotions are positive and unpleasant emotions are negative (e.g., enjoyment is positive, surprise is neutral, frustration is negative), and level of activation (e.g., anxiety is activating, boredom is deactivating; see Pekrun and Stephens, 2012).

In educational psychology, one important line of research has concerned achievement emotions, that is, emotions that are tied to achievement activities (e.g., studying) or achievement outcomes (success or failure), such as anxiety, pride, or shame. However, not all emotions triggered in educational settings are related to achievement. Notably, Pekrun and Stephens (2012) distinguished topic emotions, social emotions, as well as epistemic emotions. Topic emotions relate to the content of learning (e.g., pride when learning about the American space conquest), whereas social emotions focus on relations to others in the learning context (e.g., compassion, gratitude; Weiner, 2007). Of particular relevance to critical thinking, epistemic emotions relate to the perceived quality of knowledge and the processing of information (Pekrun and Stephens, 2012).

Muis et al. (2018) proposed that epistemic emotions arise as the result of appraisals of alignment or misalignment between the characteristics of incoming messages and individuals’ cognitive characteristics, including prior knowledge, epistemic beliefs, and epistemic aims. In the context of contending with socio-scientific issues such as climate change, vaccination, or genetically modified foods, incoming messages are likely to be characterized by knowledge claims that are complex that also include a degree of uncertainty (Levinson, 2006). For individuals seeking simple and certain answers, engaging with such content may trigger a variety of epistemic emotions such as confusion, frustration, or anxiety. However, faced with the same content, individuals who expect knowledge to be uncertain and tentative, and who see value in consulting multiple sources before coming to a conclusion, may experience curiosity and enjoyment (Muis et al., 2015). When presented with tasks that engage individuals’ beliefs about the nature of knowledge and knowing, frequently occurring epistemic emotions include surprise, curiosity, enjoyment, confusion, frustration, anxiety, and boredom (Muis et al., 2015; Pekrun et al., 2017).

Surprise is likely to occur when individuals appraise new information as unexpected (Meyer et al., 1997) or when they are unable to generate an explanation for the new information (Foster and Keane, 2015). Mildly surprising information can lead to deep processing and integration of information, whereas information that is greatly surprising can be regarded as implausible and new information may fail to be integrated (Munnich and Ranney, 2018). When information is not overly complex or perceived as relatively comprehensible, curiosity may arise. Litman (2008) proposes that epistemic curiosity arises in one of two forms: as a pleasant desire for information (i.e., interest-type curiosity), or as an unpleasant urge to obtain information to close the gap between what one knows and what one wants to know (e.g., deprivation-type curiosity; see also Loewenstein, 1994; Markey and Loewenstein, 2014). If the course of curiosity is followed, enjoyment may ensue, for instance, when validation or verification of a hypothesis is achieved (Brun and Kuenzle, 2008), or when an epistemic aim is achieved (Chinn et al., 2011; Muis et al., 2018). Confusion, on the other hand, follows from a lack of understanding when novel and complex information is perceived as incomprehensible (Muis et al., 2018). Confusion can also arise in the face of severe discrepancies or contradictions, or from disruptions of goals or sequences of action (D’Mello and Graesser, 2012). If an individual repeatedly fails to resolve the discrepancy causing confusion, frustration may arise (D’Mello and Graesser, 2012; Di Leo et al., 2019; Munzar et al., 2021). Frustration can be described as a blend of anger and disappointment and, as such, can be an activating emotion when closer to anger, or deactivating if closer to disappointment (Pekrun et al., 2002).

Another negative emotion is anxiety, which arises when a message implicates knowledge that is core to one’s identity. Individuals may begin to doubt or feel uncertain about their beliefs in a proposition, and feel that their identity is threatened (Hookway, 2008). Pekrun (2006) described anxiety as a “complex” emotion that can either benefit or hinder motivation to engage in effortful thinking. On the one hand, anxiety can reduce cognitive resources such as memory, leading to poor performance on complex or difficult tasks, as well as poor academic achievement (see Pekrun et al., 2002; Zeidner, 2014). However, for some individuals, anxiety can increase extrinsic motivation to invest effort in complex processes such as analytical and critical thinking to avoid goal-related failure. Lastly, boredom may arise when information is unchallenging or when an intense negative emotion like frustration or anxiety precipitates disengagement (D’Mello et al., 2014).

Consequences of Epistemic Emotions

Pekrun (Pekrun et al., 2002; Pekrun, 2006; Pekrun and Perry, 2014) proposed that individuals process information in emotion-congruent ways. Specifically, Pekrun and colleagues proposed that positive emotions (e.g., interest-type curiosity and enjoyment) signal that the object of judgment is valuable, leading to more positive evaluations, greater efforts to engage, more elaboration of content, and more purposeful thinking than negative emotions. On the other hand, negative emotions (e.g., frustration, anxiety, and boredom) have been related to more negative evaluations, less efforts to engage (anxiety may be an exception), less elaboration of content, and more irrelevant thinking (see Pekrun et al., 2002 for a review). Further, positive emotions have been found to facilitate holistic, intuitive, and creative ways of thinking, whereas negative emotions have been associated to more focused, detail-oriented, analytical, and rigid modes of processing information (e.g., Bless et al., 1996).

Thus, critical thinking is theorized to be facilitated by optimal levels of surprise and positive emotions such as curiosity and enjoyment and hindered by certain negative emotions such as frustration and boredom. On the other hand, other negative emotions such as anxiety and confusion may be beneficial for critical thinking: D’Mello and Graesser (2014) argued that confusion is central to complex learning activities such as problem-solving and generating cohesive arguments. As such, confusion is expected to be beneficial to critical thinking because it signals that there is something wrong with the current state of affairs, which can precipitate critical thinking. However, this expectation holds only if individuals resolve confusion when it arises (D’Mello and Graesser, 2014; Munzar et al., 2021). Indeed, as previous research has shown, when confusion is not resolved, this leads to frustration and disengagement from the task and can lower achievement outcomes (Munzar et al., 2021). Similarly, anxiety in the face of complex and conflicting information may motivate critical thinking via effortful thinking to reduce the discomfort of anxiety but may also result in a decrease in critical thinking if anxiety consumes cognitive resources (Meinhardt and Pekrun, 2003).

Empirical Evidence

To date, little theoretical and empirical work has explored how epistemic cognition relates to epistemic emotions experienced when contending with complex or conflicting information. To address this gap, Muis et al. (2015) examined relations between epistemic cognition, epistemic emotions, learning strategies—including critical thinking—and learning achievement in the context of learning about climate change. They hypothesized that individuals with more constructivist beliefs would experience more positive emotions given the consistency between the to-be-learned content and their epistemic beliefs, whereas individuals with less constructivist beliefs would experience more negative emotions given the conflicting perspectives presented to them on the causes and consequences of climate change. Results from path analyses revealed that individuals who espoused more constructivist epistemic beliefs about the justification for knowing used more critical thinking strategies, and that this relationship was mediated by curiosity: The more learners believed that knowledge is justified by systematic inquiry and integration of sources of information, the more they experienced curiosity and, in turn, the more they used critical thinking and attained greater learning achievement. They also found that surprise negatively predicted critical thinking, but surprise was not predicted by any epistemic belief dimension.

In sum, significant relations between epistemic cognition, epistemic emotions, and critical thinking are suggested in the literature. However, the studies reviewed were predominately designed to assess relations between epistemic beliefs, epistemic emotions, and critical thinking during learning; they did not instruct participants to think critically. As Greene et al. (2014) argued, the study of epistemic cognition and critical thinking should involve the need to argue for, and justify, conclusions drawn across sources and perspectives. As such, to fully understand the role of epistemic cognition and epistemic emotions in critical thinking, more research is needed wherein individuals are asked to engage in critical thinking during a complex learning task. We address this gap in the literature.

The Current Study

On the basis of theoretical and empirical considerations from Muis et al. (2015, 2018), Pekrun (Pekrun et al., 2002, 2017; Pekrun, 2006), as well as from the work of D’Mello and colleagues (D’Mello and Graesser, 2012; D’Mello et al., 2014), we propose the following hypotheses: (1) Epistemic beliefs will predict critical thinking. Specifically, more constructivist beliefs will positively predict critical thinking. (2) Epistemic beliefs will predict epistemic emotions. Specifically, more constructivist epistemic beliefs will positively predict positive epistemic emotions, including interest-type curiosity and enjoyment, and negatively predict surprise and negative emotions, including confusion, frustration, anxiety and boredom. (3) Epistemic emotions will predict critical thinking. Specifically, surprise, curiosity, enjoyment, confusion, and anxiety will positively predict critical thinking, whereas frustration and boredom will negatively predict critical thinking. (4) Epistemic emotions will mediate relations between epistemic beliefs and critical thinking.

Materials and Methods

To test these hypotheses, we designed a study that specifically embedded a task that challenged individuals to critically evaluate knowledge claims from opposing perspectives, and to take a position on the topic in the form of an argumentative essay. The topic selected was genetically modified foods. Participants first took a knowledge assessment test to assess baseline knowledge about genetically modified foods, reported their epistemic beliefs about genetically modified foods, and then read a text on genetically modified foods that was comprised of two parts. The first part of the text was informative in nature and written in the style of a refutation text to ensure that all participants would engage in essay writing with good baseline knowledge about the nature of genetically modified foods. Refutation texts address commonly held misconceptions and directly refute them by presenting correct scientific explanations (Sinatra and Broughton, 2011). The effectiveness of refutation texts for facilitating the revision of misconceptions has been well documented (see Tippett, 2010). The second part of the text was argumentative in nature and presented a series of points in favor for and against genetically modified foods. These points were supported by evidence that varied in strength and degree of certainty, but all information provided was valid. After having read the experimental text, participants wrote an argumentative essay of their choice in favor for or against genetically modified foods.

Procedure

Participants were recruited from three research-intensive universities from Eastern Canada (40.7%), Western Canada (26.5%), and the Southern United States (32.8%). Ethics approval was first obtained by the ethics review board for each participating university. To recruit participants, flyers were posted around university campuses, advertisements were posted on university websites, and subject pools from psychology courses were used. Participants provided informed consent to participate in the study and then completed a prior knowledge test and the Topic-Specfic Epistemic Beliefs Questionnaire (Bråten and Strømsø, 2009) to assess epistemic beliefs about genetically modified foods. Participants were then randomly assigned to read a version of the text that presented the advantages of genetically modified foods first (n = 102), or the disadvantages of genetically modified foods first (n = 102). After reading, participants completed the Epistemic Emotions Survey (Pekrun et al., 2017) to capture the epistemic emotions they experienced while reading. Lastly, participants composed an argumentative essay and then completed a demographics questionnaire to conclude the study. Participants were compensated for their time with $15 cash, a $10 gift card, or course credit, depending on the university from which the participant was recruited. Figure 1 provides an overview of the procedure.

FIGURE 1

Figure 1. Overview of procedure.

Participants

Two hundred four university students from three universities across Canada and the United States participated. See Table 1 for a breakdown of all demographic characteristics of the sample by gender, year in university, race, and first language spoken. No differences between groups were found on any of the variables of interest as a function of university location, gender, year in university, or first language spoken. Participants studied a variety of domains (e.g., business administration, social sciences, natural sciences, computer sciences, psychology, linguistics, and arts) and reported an average GPA of 3.24 out 4.0 (SD = 0.55). Participants from the Western Canadian institution reported significantly lower GPA (M = 2.97, SD = 0.67) than participants from the Eastern Canadian (M = 3.43, SD = 0.44) and Southern American institutions [M = 3.30, SD = 0.39; F(2, 124) = 10.02, p < 0.001]. Overall, no significant differences were observed between Canadian (M = 3.23, SD = 0.60) versus American (M = 3.30, SD = 0.39) participants in terms of reported GPA. Participants were 21.46 years of age on average (SD = 4.28).

TABLE 1

Table 1. Demographic characteristics of the sample.

Materials

Experimental Text

Participants were given a text that first presented factual information about genetically modified foods, followed by a portion that presented advantages and disadvantages of genetically modified foods. The first half of the text was adapted from Heddy et al. (2017) and focused on debunking four common misconceptions about genetically modified foods by presenting accurate scientific explanations. Erroneous conceptions included the notion that genetically modifying food is the same process as cloning, that it involves injecting hormones into a plant or animal, that it only occurs in laboratories by scientists, and that it is the product of contemporary scientific research.

The second part of the text presented four advantages of, and four criticisms against genetically modified foods. It was written by the first author and adapted from content published by the Canadian Standards Association (Whitman, 2000). To counterbalance a possible effect of text order with regard to the presentation of the advantages and disadvantages of genetically modified foods, two versions of the text were created: one version presented the advantages first, followed by the disadvantages, and the other version presented the disadvantages first, then the advantages. The text contained 1,295 words in total, including the informative and argumentative sections, with a Flesch-Kincaid index of grade 12.7 and a Flesch Reading Ease index of 37.7 (see Kincaid et al., 1975).

Prior Knowledge Test

Participants’ prior knowledge about genetically modified foods was measured with a 10-item multiple-choice test adapted from Heddy et al. (2017). Each question presented four possible choices and participants were instructed to select the best answer. Examples of items include: “Cross-pollination is considered to be a process through which plants can be… (a) genetically modified. (b) cloned. (c) hormone injected. (d) exactly replicated.” “Methods that are NOT used in producing genetically modified foods include which of the following? (a) Gene cloning methods. (b) Hormone injection. (c) Cross pollination. (d) selective pollination.” Correct answers were given a score of 1 and incorrect answers were given a score of 0. Scores were then added to create a total sum, then a percentage, which was used as an indicator of prior knowledge.

A confirmatory factor analysis (CFA) was conducted to examine the factor structure of the prior knowledge test using Mplus Version 7.11 (Muthén and Muthén, 2015). The initial model revealed a poor fit, χ² = 103.94, df = 35, p < 0.001, RMSEA = 0.05, and CFI = 0.88. An analysis of item loadings revealed low loadings for two items; therefore, these items were deleted. The final model (with the remaining eight items) resulted in a good fit, χ² = 64.14, df = 20, p < 0.01, CFI = 0.94 and RMSEA = 0.04. Cronbach’s reliability coefficient was acceptable, α = 0.79.

Epistemic Beliefs

Epistemic beliefs about genetically modified foods were measured with a version of the Topic-Specific Epistemic Beliefs Questionnaire (TSEBQ; Bråten and Strømsø, 2009) adapted to this topic. The TSEBQ comprises 24 items that participants rate on a 7-point Likert scale ranging from “strongly disagree” to “strongly agree.” Four dimensions of epistemic beliefs were measured: six items assessed beliefs about the complexity of knowledge (e.g., “Knowledge about genetic modification is primarily characterized by a large amount of detailed information”), six items assessed beliefs about the uncertainty of knowledge (e.g., “Certain knowledge about genetic modification is rare”), five items assessed beliefs about the source of knowing (e.g., “I often feel that I just have to accept that what I read about genetic modification problems can be trusted”), and seven items assessed beliefs about justification for knowing (e.g., “When I read about issues concerning genetic modification, I evaluate whether the content seems logical”).

A CFA was conducted to examine the factorial validity of scores for the instrument using Mplus7. The initial model (with 24 items) showed poor fit, χ² = 419.25, df = 246, p < 0.001, RMSEA = 0.06, and CFI = 0.78. Due to low loadings, 10 items were deleted: three items were removed from the uncertainty subscale, three from the complexity subscale, two from the source subscale, and three from the justification subscale. The final model (with 14 dimensions) resulted in good fit, χ² = 102.31, df = 71, p < 0.001, RMSEA = 0.05, and CFI = 0.93. Cronbach’s reliability coefficients were acceptable, α = 0.79 for the uncertainty subscale, α = 0.78 for the complexity subscale, α = 0.78 for the source subscale, and 0.76 for the justification subscale.

Epistemic Emotions

Epistemic emotions experienced while reading the experimental text were measured with the Epistemic Emotions Survey (EES; Pekrun et al., 2017). This questionnaire comprises 21 items that measure seven epistemic emotions, including: surprise, curiosity, enjoyment, confusion, frustration, anxiety, and boredom. Each item consisted of a single word describing one emotion, with three descriptors per emotion (e.g., “anxious,” “nervous,” and “worried” measured anxiety). Participants rated the intensity of their emotional responses to the text using a five-point Likert scale ranging from “Not at all” to “Very strong.” The scores for the descriptors of each emotion were averaged to represent each emotion. Cronbach’s reliability coefficients were acceptable, α = 0.78 for surprise; α = 0.76 for curiosity; α = 0.84 for enjoyment; α = 0.77 for confusion; α = 0.83 for frustration; α = 0.85 for anxiety; and α = 0.80 for boredom.

Essay

To assess critical thinking, we chose to measure argumentation, the second key dimension of critical thinking (Kuhn, 2018). Accordingly, participants were instructed to compose an argumentative essay of their choice in favor for or against genetically modified foods and to justify their position. Instructions were as follows: “Based on the content you just read, write a brief (2–3 paragraphs) argument for or against genetically modified foods. Explain how you came to form and justify your point of view. You can refer back to the text you read, and include your judgment of the arguments, evidence, and conclusions it presented.” Critical thinking was assessed using a coding scheme developed for this purpose.

Coding critical thinking in essays

A coding scheme was developed by the second author to assess critical thinking in argumentative essays. The coding scheme was informed by the work of Facione and Facione (2014), which outlines the development and use of a scoring rubric for evaluating critical thinking (see Table 2 for full descriptions and examples). Five elements were targeted via the coding scheme: taking a position, presenting supportive arguments in favor of a position, acknowledging an alternative perspective, evaluating the validity of claims on both sides of the issue, and integrating arguments from opposing viewpoints into a coherent perspective or conclusion. One point was attributed if participants took a position; no points were attributed if participants did not take a position. One point was attributed if participants supported their position with valid arguments, evidence, facts or reasons; no points were attributed if no arguments were presented in support of their position or if arguments were invalid. One point was attributed if participants acknowledged and presented an alternative perspective on genetically modified foods; no points were attributed if participants only presented arguments in favor of one perspective. One point was attributed if participants evaluated claims or arguments before accepting them as valid; no points were attributed if participants expediently accepted or dismissed claims or arguments without evaluation. Lastly, one point was attributed if participants reconciled or integrated perspectives; no points were attributed if the conclusion was one-sided, categorical, or failed to acknowledge the validity of any counter-argument. Points were summed to create a total score on five.

TABLE 2

Table 2. Coding scheme for critical thinking in argumentative essays.

The coding scheme was tested by the second and third authors using 31 transcripts (15% of the sample), and inter-rater reliability for the first round was established at 75%. All disagreements were resolved through discussion and were used to update the coding scheme. A second round of coding was performed with an additional 31 transcripts (new 15% of the sample), and final inter-rater reliability was established at 88%. The second author then coded the remainder of the essays.

Results

Preliminary Analyses

Prior to conducting full analyses, all variables were inspected for skewness and kurtosis. Based on Tabachnick and Fidell’s (2013) recommendations, acceptable ranges of ±3 for skewness and ±8 for kurtosis were used to investigate the relative normality of the distributions for each variable. Analyses revealed that the distributions for confusion (4.45), frustration (7.28), anxiety (3.73), and boredom (6.10) were positively skewed; however, given the nature of emotions, normal distributions for these variables are unlikely, so the variables were retained for subsequent analyses. Examination of text order (i.e., advantages of genetically modified foods first or disadvantages of genetically modified foods first) showed no or order effects on any variable. Descriptive statistics for all variables are presented in Table 3 and correlations between variables are presented in Table 4.

TABLE 3

Table 3. Descriptive statistics for variables.

TABLE 4

Table 4. Correlations between variables.

To check for univariate outliers, each variable was converted to a standardized z-score. Any z-scores exceeding critical cut-offs of ±3.3 was considered an outlier (Tabachnick and Fidell, 2013). Results revealed univariate outliers for justification (n = 2, z = −3.36 to −5.53) and frustration (n = 1, z = 3.51). Instead of deletion, all cases were retained given the values were not extreme and did not exceed more than 2% of cases for each variable (see Cohen et al., 2003). To check for multivariate outliers, Mahalanobis distances were calculated based on a χ² distribution with 12 degrees of freedom and a critical cut-off point of 32.91 (α = 0.001; see Meyers et al., 2017; Tabachnick and Fidell, 2013). No multivariate outliers were found.

Mediation Path Analysis

To test the hypothesized mediation model, we conducted a mediation analysis using Hayes (2018) PROCESS macro for SPSS, which is recommended for testing complex mediational models and maintaining high power while controlling for Type I error rates (see Hayes, 2018). Bootstrap sampling was used (with 10,000 bootstraps), which does not require assumptions of normality and which was appropriate given a few slightly skewed variables. A power analysis using G^∗Power (Faul and Erdfelder, 1992; for a full description, see Erdfelder et al., 1996) with power (1–β) set at.80 and α set at 0.05 revealed a required sample size of 218 for the present analysis. Given a sample of 204, the analysis would be underpowered. As such, we adjusted the level of the confidence intervals to 90% for the bootstrap sampling, which required a sample size of 180. The final model is depicted in Figure 2 with standardized effects.

FIGURE 2

Figure 2. Final model with standardized coefficients. Only significant paths are represented. *p < 0.05; **p < 0.01.

We first examined the total effects model, which expresses the sum of the direct and indirect effects of epistemic beliefs on critical thinking scores to determine the predictive relations between epistemic beliefs and critical thinking, independent of the effects of mediational variables. We next calculated the direct effects of epistemic beliefs on epistemic emotions, the direct effects of epistemic beliefs on critical thinking, and the indirect effects of epistemic beliefs on critical thinking via epistemic emotions. At each step, we controlled for the effects of prior knowledge.

Complexity beliefs (β = 0.16, SE = 0.06, t = 2.06, p = 0.04) and uncertainty beliefs (β = 0.14, SE = 0.07, t = 2.07, p = 0.04) were direct predictors of critical thinking. For direct effects of epistemic beliefs on epistemic emotions, complexity beliefs predicted surprise (β = -0.24, SE = 0.07, t = −3.52, p = 0.0005), enjoyment (β = 0.15, SE = 0.07, t = 2.04, p = 0.04) and confusion (β = −0.28, SE = 0.07, t = −4.02, p = 0.0001); source beliefs predicted surprise (β = −0.15, SE = 0.07, t = −2.21 p = 0.02), and enjoyment (β = −0.18, SE = 0.07, t = −2.59, p = 0.03); and justification beliefs predicted curiosity (β = 0.14, SE = 0.07, t = 2.02, p = 0.04), confusion (β = −0.14, SE = 0.07, t = −2.01, p = 0.04), and boredom (β = −0.15, SE = 0.06, t = −2.02, p = 0.04). For the direct effects of epistemic emotions on critical thinking, confusion (β = 0.24, SE = 0.10, t = 2.30, p = 0.02) and anxiety (β = 0.18, SE = 0.10, t = 2.17, p = 0.03) were significant positive predictors, and frustration (β = −0.24, SE = 0.10, t = −2.40, p = 0.01) was a significant negative predictor of critical thinking. Finally, for indirect effects of epistemic beliefs on critical thinking via epistemic emotions, results showed that the effect of complexity beliefs on critical thinking was mediated by confusion, with a point estimate of −0.07 and bias corrected bootstrapped confidence interval (90%) of −0.12 to −0.02.

Two Illustrative Cases

The following cases reflect examples of how epistemic beliefs and epistemic emotions related to critical thinking for different individuals. These cases were chosen as they represent individuals with similar demographic profiles and levels of prior knowledge about genetically modified foods, but whose epistemic beliefs and emotions as well as critical thinking skills present an interesting contrast.

Case 1

Case 1 was a 24-year-old female in the 3rd year of an environmental sciences degree with a self-reported GPA representing an academic average between 80–84% (or A-). Her prior knowledge about genetically modified foods was below average (test score = 20%). She reported epistemic beliefs that were slightly less constructivist than average on the complexity subscale (score = 3.33/7.00), less constructivist than average by more than two standard deviations on the uncertainty subscale (score = 2.83/7.00), and less constructivist than average on the source subscale by one standard deviation (score = 2.60/7.00). For epistemic emotions, she reported slightly less confusion than average (score = 1.33/5.00), slightly more frustration than average (score = 2.00/5.00), more anxiety than average by more than one standard deviation (score = 3.33/5.00), and more boredom than average by more than a standard deviation (score = 3.00/5.00).

Our analysis of Case 1’s essay indicated little critical thinking (score = 2/5) and reflected a one-sided view of genetically modified foods. Her essay included a well-positioned positive stance on genetically modified foods (“Genetically modified food is the way of the future”) as well as a few arguments in its support (“For instance, rice can be GM to have more nutrients, thus preventing millions of people from starvation” and “Already there are many third world nations that have hungry and malnourished populations. Genetically modified foods can help them by modifying their staple of food grown there.”) However, Case 1 did not identify nor engage with arguments from the opposing position. No arguments against genetically modified foods were specifically identified. Only the fact that genetically modified foods could have detrimental health effects was alluded to in a sentence that quickly dismissed the counter-argument with a statement that was justified by means of not having directly observed any opposing evidence: “Every day, there are hundreds of foods being bought in grocery stores that are GM and so far there have been no real significant downside to eating it (detrimental). In fact, I’m sure you’ve even eaten something that’s been GM this week!” Further, no conclusions were reached that hinted to an integration or reconciliation of perspectives. A conclusive statement was offered that solidified a position in favor of genetically modified foods (“Our knowledge is meant to be passed on to others so they can benefit from the fortunes that we are so lucky to have.”). Overall, Case 1 is representative of individuals with less constructivist epistemic beliefs who did not present elaborate critical thinking. Further, though prior knowledge was low, Case 1 reported little confusion. She also reported high levels of frustration, anxiety, and boredom. For Case 1, it may be the case that the presentation of opposing arguments led to more frustration and anxiety given her less constructivist beliefs about genetically modified foods. That is, consistent with Muis et al. (2018), the nature of the information presented to her was in stark contrast to her epistemic beliefs, thus triggering negative epistemic emotions. She expected knowledge about genetically modified foods to be certain and simple but was presented as uncertain and complex. This increase in frustration and anxiety may have then led her to focus solely on one side of the argument, resulting in lower performance on the task.

Case 2

Case 2 was a 24-year-old female in the 2nd year of a degree in psychology. She reported a GPA representing an academic average between 85–89% (or A). Akin to Case 1, Case 2’s prior knowledge about genetically modified foods was below average (test score = 20%). She reported epistemic beliefs that were more constructivist than average by more than one standard deviation on the uncertainty subscale (score = 5.00/7.00), more constructivist than average by more than one standard deviation on the uncertainty subscale (score = 5.83/7.00), and slightly less constructivist than average on the source subscale (score = 4.00/7.00). For epistemic emotions, she reported more confusion than average by more than one standard deviation (score = 3.00/5.00), more frustration than average by more than one standard deviation (score = 2.67/5.00), slightly more anxiety than average (score = 3.00/5.00), and slightly less boredom than average (score = 1.67/5.00).

Case 2’s essay reflected an integrated perspective on genetically modified foods. Case 2 first assumed a cautiously positive stance on genetically modified foods: “Though the use of genetically modified foods may present possible solutions to certain of the world’s problems, there is insufficient research on the matter and, more specifically, evidence supporting its proposed benefits.” She then presented some of benefits of genetically modified foods: “Genetically modified foods have been proposed to aid in addressing the many problems tied to the ever-growing population of the world, including malnutrition and land usage” and then exposed some criticism, pointing to a lack of supportive evidence, “However, these are mere propositions based on hypothetical scenarios, i.e., there is no evidence to show that certain foods can be genetically modified to provide additional vitamins and minerals - what has been proposed is a hypothetical solution.” The same pattern was repeated with the opposing perspective: Case 2 first presented arguments against genetically modified foods: “Meanwhile, a growing body of research is pointing to evidence supporting its harmful side-effects. For instance, a causal link was found between the presence of the modified B.t. corn and death of monarch butterfly caterpillars. Research has also shown that GM fed rats had digestive tracts that differed to rats fed unmodified foods”, then identified limitations, “While research on the effects of GM foods in humans is still rather limited, such animal studies are an important start.” A full reconciliation of perspectives was not reached, but a conclusion was drawn that followed the aforementioned evaluations and identified a lack of evidence as a halt to fully embracing the benefits of genetically modified foods: “Overall, the research on genetically modified foods remains inconsistent and limited. There is insufficient evidence to show that the benefits of genetically modified foods could outweigh its costs.” It may be the case that an optimal level of anxiety and confusion, combined with low boredom, motivated Case 2 to exert efforts to analyze each perspective on genetically modified foods to better understand their characteristics and nuances, resulting in observable critical thoughts.

Discussion

Socio-scientific issues such as genetically modified foods are often depicted as controversial by influencers who are either in favor or against the propositions of scientific expertise. In the face of such issues, successful critical thinking occurs when individuals purposefully decide what to believe or what to do by evaluating knowledge claims and reconciling opposing views, taking relevant evidence and context into account (Ennis, 1987; Facione, 1990). Prior theoretical and empirical work suggests that individuals’ thoughts and beliefs about the nature of knowledge and knowing play an important role in supporting critical thinking. However, little is known about the role that knowing-related emotions may play in critical thinking and the effects of epistemic cognition on such thinking. We hypothesized that epistemic cognition supports critical thinking via epistemic emotions.

This research contributes to the literature on epistemic cognition and epistemic emotions by empirically testing Muis et al. (2015) and Muis et al. (2018) model of epistemic cognition and epistemic emotions, and by providing new findings concerning relations between epistemic cognition, epistemic emotions, and critical thinking. Further, this study is the first to explore these relations in the context of an elaborate critical thinking task where participants were asked to decide what to believe about a socio-scientific issue on the basis of conflicting evidence. Specifically, results showed that a belief in complex and uncertain knowledge directly predicted critical thinking (Hypothesis 1). Complexity, source, and justification beliefs also predicted epistemic emotions, including surprise, curiosity, enjoyment, confusion, and boredom (Hypothesis 2), and epistemic emotions, confusion, frustration, and anxiety, in turn predicted critical thinking (Hypothesis 3). Lastly, confusion mediated relations between epistemic beliefs and critical thinking (Hypothesis 4). Next, we interpret each of the results described above and conclude with a discussion of limitations and directions for future research.

The Role of Epistemic Beliefs When Facing Socio-Scientific Issues

In support of our hypothesis, more constructivist epistemic beliefs about the nature of knowledge (complexity and uncertainty dimensions) significantly predicted critical thinking, indicating that the more individuals believed in complex and tentative knowledge, the more they presented support for arguments, acknowledged alternatives, evaluated claims, and drew balanced conclusions. However, epistemic beliefs about the nature of knowing (beliefs about the sources of, and justification for knowing) were not significantly related to critical thinking. It should be mentioned that it is frequent in epistemic belief research that not all belief dimensions are salient in a given situation, depending on the nature of the task (Hammer and Elby, 2002; Greene et al., 2010). Similar to this study, Strømsø et al. (2011) examined relations between epistemic beliefs and undergraduate students’ evaluations of documents’ trustworthiness and found that source beliefs significantly predicted evaluation of conflicting claims, but justification beliefs did not contribute significantly to trustworthiness scores.

For our study, three dimensions of epistemic beliefs were found to have direct effects on five epistemic emotions. In particular, in line with hypotheses, the more individuals believed that knowledge about genetically modified foods is complex, the less likely there were to experience surprise and confusion, and the more likely they were to experience enjoyment. This supports the notion that epistemic beliefs shape individuals’ assumptions about the nature of knowledge (Bromme et al., 2010), such that those who expected knowledge about genetically modified foods to be simple may have experienced dissonance related to the complex nature of information presented in the text. Individuals who expected knowledge to be complex, when presented with conflicting information, were not surprised by this conflict nor were they confused about the conflicting information. Moreover, consistent with hypotheses, more constructivist complexity beliefs predicted more enjoyment when reading about advantages and disadvantages of genetically modified foods.

Following Muis et al. (2015) and Muis et al. (2018) model of epistemic cognition and epistemic emotions, we hypothesized that enjoyment would stem from an alignment between epistemic beliefs that are congruent with the nature of science (i.e., more constructivist epistemic beliefs) and the epistemic nature of the material presented. Similarly, Franco et al. (2012) found that when individuals’ epistemic beliefs are consistent with the knowledge representations in complex learning material, they perform better on various measures, including deep processing of information, text recall, and changes in misconceptions. However, Muis et al. (2018) suggested that epistemic emotions have more antecedents than were measured here, including perceptions of control and task value, as well as information novelty and complexity. They argued that if an individual with more constructivist epistemic beliefs has low perceived control or assigns little value to the task at hand, then he or she may experience lower levels of enjoyment. This suggests that epistemic beliefs alone cannot fully predict the type of epistemic emotions that are likely to arise in a given situation. As such, to fully understand the relationship between epistemic cognition and epistemic emotions more broadly, future work should include other epistemic emotion antecedents and take further contextual elements into account.

Additionally, those who viewed personal interpretations and judgments as the main sources of knowledge about genetically modified foods experienced less surprise but also less enjoyment during learning when reading contradictory perspectives about the value and usefulness of genetically modified foods. This result is consistent with findings from Strømsø et al. (2011) who found that the more students viewed the self as a meaning maker, the less they trusted texts written by climate change experts. Similarly, Kardash and Scholes (1996) found that the less students believed in external authority as a source of knowledge, the stronger their opinions about the HIV-AIDS relationship.

It could be the case that individuals who believe that knowledge resides within the self (and who have low prior knowledge) also prefer to fall back on their own opinions and find it less enjoyable to have to consider the point of view of others. Traditionally, the belief that knowledge originates from external authorities has been viewed as “naïve,” whereas the conception of self as a knower has been viewed as “sophisticated” (Hofer and Pintrich, 1997). However, researchers have called into question the assumption that more constructivist beliefs are better to espouse in all situations (see Bromme et al., 2008; Greene et al., 2010; Greene and Yu, 2014). Indeed, when novices face a complex topic such as genetically modified foods, it may be adaptive to assume that experts are trustworthy and to balance one’s own judgments with reliance on external expert sources.

Moreover, when individuals are presented with conflicting information about a topic, it is beneficial to evaluate and integrate evidence and arguments. That is, individuals who believed that knowledge is justified through a process of critical evaluation and integration of information experienced more curiosity and less confusion and boredom compared to individuals who believed in an unquestioning reliance on authorities. In the case of the texts presented to participants in this study, authorities reported both pros and cons about genetically modified foods. Under this condition, individuals are likely more confused given that they may be uncertain as to which authority to trust, may find the task too challenging, and then experience greater boredom. However, as previous research has shown (D’Mello et al., 2014), confusion can be beneficial for learning by increasing critical thinking. We describe relations between emotions and critical thinking next.

The Role of Epistemic Emotions in Critical Thinking

Consistent with the contention that confusion can be beneficial for complex cognitive tasks, confusion was found to be a positive predictor of critical thinking. Also consistent with hypotheses, confusion was negatively predicted by complexity and justification beliefs and, as such, fully mediated relations between these beliefs and critical thinking. Although the full mediation effect seems to suggest that more constructivist beliefs are detrimental to critical thinking via decreased levels of confusion, we suggest that effects revealed here are more complex than they appear. It might be the case that compared to individuals with less constructivist epistemic beliefs, those who espouse more constructivist beliefs experience less confusion related to the complex nature of genetically modified foods knowledge, but nevertheless perceive discrepancies between perspectives that can trigger lower levels of confusion associated with beneficial effects. Indeed, philosophers such as Morton (2010) and Elgin (2008) have argued that epistemic emotions such as surprise and confusion can draw attention to the object of the emotion, which can lead to deep processing of information as well as metacognitive self-regulation (Muis et al., 2015). Moreover, two of the epistemic belief dimensions directly positively predicted critical thinking. As such, it may be the case that individuals with more constructivist epistemic beliefs experience less confusion, but still directly engage in critical thinking given that they believe that knowledge must be critically evaluated and that perspectives must be weighed before coming to a specific conclusion on an issue.

In contrast to confusion, frustration was found to be a negative predictor of critical thinking. Frustration is an intense negative emotion that can overtake the cognitive system (Rosenberg, 1998), and is linked to a reduction of effortful thinking and an increase of rigid and shallow processing of information (see Pekrun et al., 2011; Pekrun and Stephens, 2012). D’Mello and Graesser (2012) proposed that frustration can lead to boredom and ultimately, disengagement from task. Moreover, we observed a significant positive relationship between anxiety and critical thinking, suggesting that anxiety may be beneficial for critical thinking. This result was expected and is consistent with Muis et al.’s (2015) results, who also noted a significant positive path from anxiety to critical thinking. In the present study, anxiety was unrelated to epistemic beliefs but may have been related to epistemic aims such as to understand the content or find the truth about genetically modified foods. Measuring epistemic aims as antecedents of epistemic emotions will be an important avenue to understand the conditions under which anxiety can benefit critical thinking, and those under which it does not.

In terms of positive emotions, in this study, we did not find significant predictive relationships between enjoyment and critical thinking. Therefore, the current results do not replicate prior work by Muis et al. (2015), who found curiosity to predict critical thinking. Muis et al. (2018) proposed that curiosity and confusion are similar in that they both result from surprise triggered by dissonance, incongruity, or uncertainty. They proposed that the complexity of information or of a task predicts whether curiosity or confusion follows surprise. Specifically, they argued that when complexity is high, surprise may turn into confusion, whereas curiosity is more likely to ensue in cases where discrepancies can be easily revolved. In the current study, it appears that curiosity and confusion highly co-occurred. More research is needed to better understand how individuals experience curiosity and confusion when trying to determine what is true or what to believe about a complex and controversial topic.

Overall, the current study provided support for many of the predictions posited in the epistemic cognition and emotion literature, yet also provided new insights into the epistemic and affective nature of critical thinking. Specifically, the notion that more constructivist epistemic cognition promotes critical thinking was generally supported, as was the contention that epistemic emotions mediate relations between epistemic cognition and cognitive processes. Further, results supported the idea that milder forms of negative emotions such as anxiety and confusion can be beneficial for critical thinking, whereas intense activating negative emotions (i.e., activating forms of frustration) are detrimental for critical thinking. However, results also challenged the assumptions that positive emotions are required for critical thinking to occur. Lastly, our results challenge dominant conceptions about beliefs in the self as the primary source of knowledge as being beneficial for critical thinking. Our counter-hypothetical results provide additional support for the idea that there is a need to reconsider and reinvestigate how individuals productively conceive of and justify knowledge (see Greene et al., 2008; Chinn et al., 2011; Greene and Yu, 2014). Overall, findings from the current study support the notion that critical thinking is not necessarily something that feels good (Danvers, 2016), yet suggest that espousing more constructivist beliefs about the nature of knowledge may benefit critical thinking by tampering certain difficult emotions and supporting the use of critical thinking.

Educational Implications

The results obtained in the present study have several implications for educational interventions aimed at increasing critical thinking about socio-scientific issues. First, findings support the notion that knowledge- and knowing-related issues should be highlighted and discussed in educational settings, with the aim of developing more constructivist forms of epistemic cognition. Notably, discussions surrounding the complex and tentative nature of scientific knowledge may be beneficial to shaping individuals’ expectations about the issues they will be called upon to reflect and act on during their lifetime. Barnett (2004), a prominent philosopher of higher education, has described the mission of university education as preparing students for a complex and uncertain future: For individuals to prosper, make decisions, and come to a position of security amid multiple interpretations, individuals must come not only to learn for uncertainty, but to learn to live with uncertainty. Barnett contends that no risk-free curricular approach can achieve this; instead, he calls for a curriculum that aims at educational transformation through exposure to dilemmas and uncertainties. This may include, for instance, confronting students with the limits of knowing in a field and with the limitations of the field as such. In addition to uncertainty- and complexity-focused curricula, Muis et al. (2016) proposed that to achieve epistemic change, epistemic climates are needed that involve constructivist pedagogical approaches (e.g., inquiry-based learning, apprenticeship, collaborative learning, knowledge building, and communities of practice), decentralized authority structures, open-ended assessment practices, and appropriate levels of teacher support, as students experience the sometimes difficult process of belief change.

Second, findings from the present study suggest that to develop critical thinking about socio-scientific issues, learning environments should be supportive of students’ emotional responses. In particular, for students with less constructivist epistemic cognition, being exposed to complex and conflicting information may trigger surprise, confusion, and frustration. We argue that such emotions should be welcomed without judgment by teachers and peers, and that these emotional experiences should be normalized (Di Leo and Muis, 2020). Further, teachers should discuss their own epistemic emotions and model appropriate emotion regulation strategies (Gross, 2014). Related to confusion, students may have a tendency to want to avoid confusion by seeking out tasks with minimal intellectual challenges (situation selection), seeking help when challenged (situation modification), or intentionally ignoring or misattributing the cause of discrepant events to avoid confusion (reappraisal; D’Mello and Graesser, 2014; Gross, 2014; Harley et al., 2019). However, teachers can discuss the drawback of these strategies, and further suggest and model a different set of emotion regulation strategies, including choosing to engage in tasks that are intellectually challenging (situation selection), open up to perspectives that do not at first flatter their preferred position (situation modification), and help students build competencies for critical reflection (competence enhancement). By reinforcing the latter strategies, students may become what Clifford (1988) describes as “academic risk takers,” who are more tolerant to uncertainty and failure.

Third and relatedly, given observed relations between beliefs, confusion and critical thinking, we suggest that students with less constructivist epistemic beliefs may benefit from learning materials that trigger mild confusion, but without giving way to frustration. To this end, D’Mello and Graesser (2012) suggest pedagogical practices where misconceptions are exposed, where complexity is embraced, and where less cohesive texts and lectures replace the polished deliveries of textbooks and formal lectures. However, to avoid confusion turning into frustration or disengagement (D’Mello and Graesser, 2012; Munzar et al., 2021), teachers should support the development of students’ critical thinking skills and resolution strategies by scaffolding and modeling these abilities (Muis and Duffy, 2013; Di Leo and Muis, 2020), so that students become able to productively engage with confusion-inducing materials, to the benefit of deep and critical thinking.

Finally, it is also important to note that all students could benefit from being taught how to write argumentative essays; the task we used to capture the output dimension of critical thinking (Kuhn, 2018). As previous research has shown, undergraduate and graduate students typically perform below the expected level for argumentative essay writing tasks (Kellogg and Whiteford, 2009). Our sample was no different, with an average of just over 53%. Clearly, beyond focusing on epistemic cognition and epistemic emotions in the classroom, students could benefit from direct instruction on argumentative writing. One method that has been effective in improving students’ argumentative essay writing is via scaffolding through adaptive fading (Noroozi et al., 2018), and worked examples and peer feedback (Latifi et al., 2019; Noroozi and Hatami, 2019; Valero Haro et al., 2019). Indeed, as Muis (2007) argued, it may be the case the teaching students these skills may also help to improve their epistemic cognition. Of course, one could also argue that we measured only one key dimension of critical thinking. Future research should also measure the inquiry dimension of critical thinking to assess whether results from our study replicate. Importantly, relations between constructs may differ depending on how critical thinking is measured. We address limitations next.

Limitations and Future Directions

Several concerns may limit the results presented herein. First, the analysis used correlational associations of the study variables over time but did not experimentally manipulate the predictor variables. As such, future research should complement the approach used here with experimental studies. However, this may be easier to do with emotions, which can to some extent be manipulated experimentally, than with more stable epistemic beliefs. A second limitation concerns the rubric employed to capture critical thinking in essays. Specifically, we opted for a quantitative approach to coding critical thinking by attributing one point for the presence of each component of critical thinking. However, a weighted coding scheme or a holistic rubric are two other modes of critical thinking assessment that include qualitative elements of analysis that could have yielded different results. Third, epistemic cognition and epistemic emotions were measured via self-report, which also have inherent limitations (see, for example, DeBacker et al., 2008). Therefore, future research is needed to replicate the findings presented here using alternative methods, which we delineate below.

The current findings have important implications for future research on epistemic cognition and epistemic emotions. Specifically, to fully understand how epistemic cognition supports critical thinking, future research should explore the role that other facets of epistemic cognition play in mediating this relationship. For instance, how do individuals’ knowledge of epistemic strategies shape critical thinking, and do these abilities influence the arousal of epistemic emotions in the face of complex and conflicting information? And how might epistemic aims moderate these relations? Prior work has shown that these other epistemic facets play a significant role in epistemic emotion arousal and researchers have called for more research on epistemic cognition that conceptualize and operationalize the construct beyond the sole notion of epistemic beliefs (Greene et al., 2016).

Lastly, in light of the findings revealed herein, we contend that one important avenue for future work will be to investigate how different intensities of positive, neutral, and negative epistemic emotions influence information processing and critical thinking. To this end, we believe that the self-report measurement of epistemic cognition and emotions can be complemented by and triangulated with trace data collected by think-aloud or emote-aloud protocols (e.g., Craig et al., 2008; Di Leo and Muis, 2020), physiological measures of emotions such as analysis of facial expression, electrocardiograms, and galvanic skin responses (Azevedo et al., 2013; D’Mello et al., 2014), and qualitative work. In sum, by broadening conceptual horizons and employing advanced methodologies, we believe that future research will provide a rich portrait of the ways in which epistemic cognition and epistemic emotions support critical thinking.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by McGill University Research Ethics Board. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

KM developed the larger program of study, designed the materials and research questions, cleaned and analyzed the data, and wrote the manuscript. MC helped develop the research questions, collected the data, cleaned the data, developed the coding scheme for the essays, coded the essays, analyzed the data, and wrote the manuscript. CD helped collect the data, coded the essays, and helped in writing the manuscript. KL helped collect the data and wrote the manuscript. All authors contributed to the article and approved the submitted version.

Funding

Funding for this work was provided by a grant to KM from the Social Sciences and Humanities Research Council of Canada (435-2014-0155), and from the Canada Research Chair program. Correspondence concerning this article can be addressed to KM, Department of Educational and Counselling Psychology, Faculty of Education, McGill University, 3,700 McTavish Street, Montreal, QC, H3A 1Y2, or via email at krista.muis@mcgill.ca.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

1. ^ Feelings can be emotional or non-emotional. Non-emotional feelings include physiologically derived feelings like pain, hunger, or thirst, as well as cognitive and metacognitive feelings such as judgments of knowing or learning. Emotions compose feelings but also include other components as noted in the definition.

References

Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., et al. (2008). Instructional interventions affecting critical thinking skills and dispositions: A stage 1 meta-analysis. Rev. Educat. Res. 78, 1102–1134. doi: 10.3102/0034654308326084

CrossRef Full Text | Google Scholar

Alexander, P. A. (2014). Thinking critically and analytically about critical-analytic thinking: An introduction. Educat. Psychol. Rev. 26, 469–476. doi: 10.1007/s10648-014-9283-1

CrossRef Full Text | Google Scholar

Arum, R., and Roksa, J. (2011). Academically adrift: Limited learning on college campuses. Chicago, IL: University of Chicago Press.

Google Scholar

Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-Behnagh, R., Bouchet, F., et al. (2013). “Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems,” In International handbook of metacognition and learning technologies. eds R. Azevedo and V. Aleven (New York, NY: Springer), 427–449. doi: 10.1007/978-1-4419-5546-3_28

CrossRef Full Text | Google Scholar

Bailin, S., and Siegel, H. (2003). “Critical thinking,” in The Blackwell guide to the philosophy of education, eds N. Blake, P. Smeyers, R. Smith, and P. Standish (Oxford, UK: Blackwell Publishing), 181–193.

Google Scholar

Barnett, R. (2004). Learning for an unknown future. Higher Educat. Res. Dev. 23, 247–260. doi: 10.1080/0729436042000235382

CrossRef Full Text | Google Scholar

Barzilai, S., and Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning Instruct. 36, 86–103. doi: 10.1016/j.learninstruc.2014.12.003

CrossRef Full Text | Google Scholar

Barzilai, S., and Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognit. Instruct. 30, 39–85. doi: 10.1080/07370008.2011.636495

CrossRef Full Text | Google Scholar

Barzilai, S., and Zohar, A. (2014). Reconsidering personal epistemology as metacognition: A multifaceted approach to the analysis of epistemic thinking. Educat. Psychol. 49, 13–35. doi: 10.1080/00461520.2013.863265

CrossRef Full Text | Google Scholar

Bless, H., Clore, G. L., Schwarz, N., Golisano, V., Rabe, C., and Wölk, M. (1996). Mood and the use of scripts: Does a happy mood really lead to mindlessness? J. Personal. Soc. Psychol. 71, 665–679. doi: 10.1037/0022-3514.71.4.665

PubMed Abstract | CrossRef Full Text | Google Scholar

Bråten, I. (2016). “Epistemic cognition interventions,” in Handbook of epistemic cognition, eds J. A. Greene, W. A. Sandoval, and I. Bråten (New York, NY: Routledge), 360–371.

Google Scholar

Bråten, I., and Strømsø, H. I. (2009). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discour. Proc. 47, 1–31. doi: 10.1080/01638530902959646

CrossRef Full Text | Google Scholar

Bråten, I., Anmarkrud, Ø, Brandmo, C., and Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning Instruct. 30, 9–24. doi: 10.1016/j.learninstruc.2013.11.002

CrossRef Full Text | Google Scholar

Bromme, R., Kienhues, D., and Stahl, E. (2008). “Knowledge and epistemological beliefs: An intimate but complicate relationship,” in Knowing, knowledge and beliefs: Epistemological studies across diverse cultures, ed. M. S. Khine (New York, NY: Springer), 423–441. doi: 10.1007/978-1-4020-6596-5_20

CrossRef Full Text | Google Scholar

Bromme, R., Pieschl, S., and Stahl, E. (2010). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological beliefs and metacognition. Metacognit. Learning 5, 7–26. doi: 10.1007/s11409-009-9053-5

CrossRef Full Text | Google Scholar

Brun, G., and Kuenzle, D. (2008). “A new role for emotions in epistemology?,” in Epistemology and emotions, eds G. Brun, U. Dog̃uog̃lu, and D. Kuenzle (Aldershot: Ashgate), 1–32.

Google Scholar

Chinn, C. A., Buckland, L. A., and Samarapungavan, A. L. A. (2011). Expanding the dimensions of epistemic cognition: Arguments from philosophy and psychology. Educat. Psychol. 46, 141–167. doi: 10.1080/00461520.2011.587722

CrossRef Full Text | Google Scholar

Chiu, Y. L., Liang, J. C., and Tsai, C. C. (2013). Internet-specific epistemic beliefs and self-regulated learning in online academic information searching. Metacognit. Learning 8, 235–260. doi: 10.1007/s11409-013-9103-x

CrossRef Full Text | Google Scholar

Claris, L., and Riley, D. (2012). Situation critical: critical theory and critical thinking in engineering education. Engine. Stud. 4, 101–120. doi: 10.1080/19378629.2011.649920

CrossRef Full Text | Google Scholar

Clifford, M. M. (1988). Failure tolerance and academic risk−taking in ten−to twelve−year−old students. Br. J. Educat. Psychol. 58, 15–27. doi: 10.1111/j.2044-8279.1988.tb00875.x

CrossRef Full Text | Google Scholar

Cohen, J., Cohen, P., West, S. G., and Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences, 3rd Edn. Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

Google Scholar

Craig, S. D., D’Mello, S., Witherspoon, A., and Graesser, A. (2008). Emote aloud during learning with AutoTutor: Applying the Facial Action Coding System to cognitive–affective states during learning. Cognit. Emot. 22, 777–788. doi: 10.1080/02699930701516759

CrossRef Full Text | Google Scholar

D’Mello, S. K., and Graesser, A. C. (2012). Dynamics of affective states during complex learning. Learning Instruct. 22, 145–157. doi: 10.1016/j.learninstruc.2011.10.001

CrossRef Full Text | Google Scholar

D’Mello, S. K., and Graesser, A. C. (2014). “Confusion,” in Handbook of emotions in education, eds R. Pekrun and L. Linnenbrink-Garcia (New York: Routledge), 289–310.

Google Scholar

D’Mello, S., Lehman, B., Pekrun, R., and Graesser, A. (2014). Confusion can be beneficial for learning. Learning Instruct. 29, 153–170. doi: 10.1016/j.learninstruc.2012.05.003

CrossRef Full Text | Google Scholar

Danvers, E. C. (2016). Criticality’s affective entanglements: rethinking emotion and critical thinking in higher education. Gender Educat. 28, 282–297. doi: 10.1080/09540253.2015.1115469

CrossRef Full Text | Google Scholar

DeBacker, T. K., Crowson, H. M., Beesley, A. D., Thoma, S. J., and Hestevold, N. L. (2008). The challenge of measuring epistemic beliefs: An analysis of three self-report instruments. J. Exp. Educat. 76, 281–312. doi: 10.3200/jexe.76.3.281-314

CrossRef Full Text | Google Scholar

Dewey, J. (1933). Philosophy and civilization. Boston: D.C. Heath & Co.

Google Scholar

Di Leo, I., and Muis, K. R. (2020). Confused, now what? A cognitive-emotional strategy training intervention for elementary students during mathematics problem solving. Contemporary Educat. Psychol. 62:101876.

Google Scholar

Di Leo, I., Muis, K. R., Singh, C., and Psaradellis, C. (2019). Curiosity. Confusion? Frustration! The role and sequencing of emotions during mathematics problem solving. Contempor. Educat. Psychol. 58, 121–137. doi: 10.1016/j.cedpsych.2019.03.001

CrossRef Full Text | Google Scholar

Elgin, C. Z. (2008). “Emotion and understanding,” in Epistemology and emotions, eds G. Brun, U. Doguoglu, and D. Kuenzle (Aldershot: Ashgate), 33–50.

Google Scholar

Ellsworth, P. C. (2013). Appraisal theory: Old and new questions. Emot. Rev. 5, 125–131. doi: 10.1177/1754073912463617

CrossRef Full Text | Google Scholar

Ennis, R. (1987). “A taxonomy of critical thinking dispositions and abilities,” in Teaching thinking skills: Theory and practice, eds J. Baron and R. Sternberg (New York, NY: Freeman), 9–26.

Google Scholar

Ennis, R. H. (2018). Critical thinking across the curriculum: A vision. Topoi 37, 165–184. doi: 10.1007/s11245-016-9401-4

CrossRef Full Text | Google Scholar

Erdfelder, E., Faul, F., and Buchner, A. (1996). GPOWER: A general power analysis program. Behav. Res. Methods Instruments Comput. 28, 1–11. doi: 10.3758/bf03203630

CrossRef Full Text | Google Scholar

Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (Research findings and recommendations). Newark, DE: American Philosophical Association.

Google Scholar

Facione, P. A., and Facione, N. C. (2014). The holistic critical thinking scoring rubric: A tool for developing and evaluating critical thinking. California: California Academic Press.

Google Scholar

Faul, F., and Erdfelder, E. (1992). GPOWER: A priori, post-hoc, and compromise power analyses for MS-DOS. Bonn: Bonn University.

Google Scholar

Foster, M. I., and Keane, M. T. (2015). Why some surprises are more surprising than others: Surprises as a metacognitive sense of explanatory difficulty. Cognit. Psychol. 81, 74–116. doi: 10.1016/j.cogpsych.2015.08.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Franco, G. M., Muis, K. R., Kendeou, P., Wang, X., Ranellucci, J., and Sampasivam, L. (2012). Examining the influences of epistemic beliefs and knowledge representations on cognitive processing and conceptual change when learning physics. Learning Instruct. 22, 62–77. doi: 10.1016/j.learninstruc.2011.06.003

CrossRef Full Text | Google Scholar

Gordon, J. (2000). Congruency in defining critical thinking by nurse educators and non-nurse scholars. J. Nurs. Educat. 39, 340–351.

Google Scholar

Greene, J. A., and Yu, S. B. (2014). Modeling and measuring epistemic cognition: A qualitative re-investigation. Contemp. Educat. Psychol. 39, 12–28. doi: 10.1016/j.cedpsych.2013.10.002

CrossRef Full Text | Google Scholar

Greene, J. A., Azevedo, R., and Torney-Purta, J. (2008). Modeling epistemic and ontological cognition: Philosophical perspectives and methodological directions. Educat. Psychol. 43, 142–160.

Google Scholar

Greene, J. A., Muis, K. R., and Pieschl, S. (2010). The role of epistemic beliefs in students’ self-regulated learning with computer-based learning environments: Conceptual and methodological issues. Educat. Psychol. 45, 245–257. doi: 10.1080/00461520.2010.515932

CrossRef Full Text | Google Scholar

Greene, J. A., Sandoval, W. A., and Bråten, I. (2016). Handbook of epistemic cognition. New York, NY: Routledge.

Google Scholar

Greene, J. A., Yu, S. B., and Copeland, D. Z. (2014). Measuring critical components of digital literacy and their relationships with learning. Comput. Educat. 76, 55–69. doi: 10.1016/j.compedu.2014.03.008

CrossRef Full Text | Google Scholar

Gross, J. J. (2014). “Emotion regulation: Conceptual and empirical foundations,” in Handbook of emotion regulation, 2nd Edn, ed. J. J. Gross (New York, NY: Guilford), 3–20.

Google Scholar

Halpern, D. F. (2014). Thought and Knowledge, 5th Edn. London: Psychology Press.

Google Scholar

Hammer, D., and Elby, A. (2002). “On the form of personal epistemology,” in Personal epistemology: The psychology of beliefs about knowledge and knowing, eds B. K. Hofer and P. R. Pintrich (Mahwah, NJ: Lawrence Erlbaum Associates), 169–190.

Google Scholar

Harley, J. M., Pekrun, R., Taxer, J. L., and Gross, J. J. (2019). Emotion regulation in achievement situations: An integrated model. Educat. Psychol. 54, 106–126. doi: 10.1080/00461520.2019.1587297

CrossRef Full Text | Google Scholar

Hayes, A. F. (2018). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach, 2nd Edn. New York, NY: Guilford Press.

Google Scholar

Heddy, B. C., Danielson, R. W., Sinatra, G. M., and Graham, J. (2017). Modifying knowledge, emotions, and attitudes regarding genetically modified foods. J. Exp. Educat. 85, 513–533. doi: 10.1080/00220973.2016.1260523

CrossRef Full Text | Google Scholar

Hofer, B. K., and Bendixen, L. D. (2012). “Personal epistemology: Theory, research, and future directions,” in APA Educational Psychology Handbook, Vol. 1, eds K. R. Harris, S. Graham, and T. Urdan (Washington, DC: American Psychological Association), 227–256. doi: 10.1037/13273-009

CrossRef Full Text | Google Scholar

Hofer, B. K., and Pintrich, P. R. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Rev. Educat. Res. 6, 88–140. doi: 10.3102/00346543067001088

CrossRef Full Text | Google Scholar

Hookway, C. (2008). “Epistemic immediacy, doubt and anxiety: On a role for affective states in epistemic evaluation,” in Epistemology and emotions, eds G. Brun, U. Doguoglu, and D. Kuenzle (Aldershot, UK: Ashgate), 51–65.

Google Scholar

Huber, C. R., and Kuncel, N. R. (2015). Does college teach critical thinking? A meta-analysis. Rev. Educat. Res. 86:5917.

Google Scholar

Hyytinen, H., Holma, K., Toom, A., Shavelson, R. J., and Lindblom-Ylänne, S. (2014). The complex relationship between students’ critical thinking and epistemological beliefs in the context of problem solving. Frontline Learning Res. 2, 1–25. doi: 10.14689/ejer.2017.72.11

CrossRef Full Text | Google Scholar

Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.

Google Scholar

Kardash, C. M., and Scholes, R. J. (1996). Effects of preexisiting beliefs, epistemological beliefs, and need for cognition on interpretation of controversial issues. J. Educat. Psychol. 88:260. doi: 10.1037/0022-0663.88.2.260

CrossRef Full Text | Google Scholar

Kellogg, R. T., and Whiteford, A. P. (2009). Training advanced writing skills: The case for deliberate practice. Educat. Psychol. 44, 250–266. doi: 10.1080/00461520903213600

CrossRef Full Text | Google Scholar

Kincaid, J. P., Fishburne, R., Rogers, R. L., and Chissom, B. S. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease formula) for Navy enlisted personnel. Millington, TN: Chief of Naval Training.

Google Scholar

King, P. M., and Kitchener, K. S. (2002). “The reflective judgment model: Twenty years of research on epistemic cognition,” in Personal epistemology. The psychology of beliefs about knowledge and knowing, eds B. K. Hofer and P. R. Pintrich (Mahwah, NJ: Erlbaum), 37–62.

Google Scholar

Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press.

Google Scholar

Kuhn, D. (1999). A developmental model of critical thinking. Educat. Researcher 28, 16–46. doi: 10.3102/0013189x028002016

CrossRef Full Text | Google Scholar

Kuhn, D. (2018). A role for reasoning in a dialogic approach to critical thinking. Topoi 37, 121–128. doi: 10.1007/s11245-016-9373-4

CrossRef Full Text | Google Scholar

Kuhn, D., and Crowell, A. (2011). Dialogic argumentation as a vehicle for developing young adolescents’ thinking. Psychol. Sci. 22, 545–552. doi: 10.1177/0956797611402512

PubMed Abstract | CrossRef Full Text | Google Scholar

Kuhn, D., Hemberger, L., and Khait, V. (2015). Argue with me: argument as a path to developing students’ thinking and writing (Second). London, UK: Routledge.

Google Scholar

Kurfiss, J. G. (1988). Critical thinking: Theory, research, practice, and possibilities (Ashe-Eric Higher Education Report No. 2). Washington, DC: Association for the Study of Higher Education.

Google Scholar

Latifi, S., Noroozi, O., Hatami, J., and Biemans, H. J. (2019). How does online peer feedback improve argumentative essay writing and learning? Innovat. Educat. Teaching Int. 2019, 1–12. doi: 10.1080/14703297.2019.1687005

CrossRef Full Text | Google Scholar

Levinson, R. (2006). Towards a theoretical framework for teaching controversial socio−scientific issues. Int. J. Sci. Educat. 28, 1201–1224. doi: 10.1080/09500690600560753

CrossRef Full Text | Google Scholar

Litman, J. A. (2008). Interest and deprivation factors of epistemic curiosity. Personal. Individ. Differ. 44, 1585–1595. doi: 10.1016/j.paid.2008.01.014

CrossRef Full Text | Google Scholar

Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychol. Bull. 116, 75–98. doi: 10.1037/0033-2909.116.1.75

CrossRef Full Text | Google Scholar

Markey, A., and Loewenstein, G. (2014). “Curiosity,” in International handbook of emotion in education, eds R. Pekrun and L. Linnenbrink-Garcia (New York, NY: Routledge), 228–245.

Google Scholar

Mason, L., and Boscolo, P. (2004). Role of epistemological understanding and interest in interpreting a controversy and in topic-specific belief change. Contemp. Educat. Psychol. 29, 103–128. doi: 10.1016/j.cedpsych.2004.01.001

CrossRef Full Text | Google Scholar

Mason, L., and Scirica, F. (2006). Prediction of students’ argumentation skills about controversial topics by epistemological understanding. Learning Instruct. 16, 492–509. doi: 10.1016/j.learninstruc.2006.09.007

CrossRef Full Text | Google Scholar

McGinnis, D. (2016). Epistemological orientations and evidence evaluation in undergraduates. Thinking Skills Creativity 19, 279–289. doi: 10.1016/j.tsc.2016.01.002

CrossRef Full Text | Google Scholar

Meinhardt, J., and Pekrun, R. (2003). Attentional resource allocation to emotional events: An ERP study. Cognit. Emot. 17, 477–500. doi: 10.1080/02699930244000039

PubMed Abstract | CrossRef Full Text | Google Scholar

Meyer, W. U., Reisenzein, R., and Schützwohl, A. (1997). Toward a process analysis of emotions: The case of surprise. Motivat. Emot. 21, 251–274.

Google Scholar

Meyers, L. S., Gamest, G., and Guarino, A. J. (2017). Applied multivariate research: Design and interpretation, 3rd Edn. Thousand Oaks, CA: SAGE Publications.

Google Scholar

Morton, A. (2010). “Epistemic emotions,” in The Oxford handbook of philosophy of emotion, ed. P. Goldie (Oxford: Oxford University Press), 385–399.

Google Scholar

Muis, K. R. (2007). The role of epistemic beliefs in self-regulated learning. Educ. Psychol. 42, 173–190. doi: 10.1080/00461520.2010.515932

CrossRef Full Text | Google Scholar

Muis, K. R., and Duffy, M. C. (2013). Epistemic climate and epistemic change: Instruction designed to change students’ epistemic beliefs and learning strategies and improve achievement. J. Educat. Psychol. 105, 213–225. doi: 10.1037/a0029690

CrossRef Full Text | Google Scholar

Muis, K. R., Pekrun, R., Sinatra, G. M., Azevedo, R., Trevors, G., Meier, E., et al. (2015). The curious case of climate change: Testing a theoretical model of epistemic beliefs, epistemic emotions, and complex learning. Learning Instruct. 39, 168–183. doi: 10.1016/j.learninstruc.2015.06.003

CrossRef Full Text | Google Scholar

Muis, K. R., Trevors, G., and Chevrier, M. (2016). “Epistemic climate for epistemic change,” in Handbook of epistemic cognition, eds J. Greene, B. Sandoval, and I. Bråten (London: Routledge), 331–359.

Google Scholar

Muis, R. R., Chevrier, M., and Singh, C. (2018). The role of epistemic emotions in personal epistemology and self-regulated learning. Educat. Psychol. 53, 165–184. doi: 10.1080/00461520.2017.1421465

CrossRef Full Text | Google Scholar

Munnich, E., and Ranney, M. A. (2018). Learning from surprise: Harnessing a metacognitive surprise signal to build and adapt belief networks. Topics Cognit. Sci. 11, 164–177.

Google Scholar

Munzar, B., Muis, K. R., Denton, C., and Losenno, K. (2021). Elementary students’ cognitive and affective responses to impasses during mathematics problem-solving. J. Educat. Psychol. 113, 104–124. doi: 10.1037/edu0000460

CrossRef Full Text | Google Scholar

Muthén, L. K., and Muthén, B. O. (2015). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén.

Google Scholar

Niu, L., Behar-Horenstein, L. S., and Garvan, C. W. (2013). Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educat. Res. Rev. 9, 114–128. doi: 10.1016/j.edurev.2012.12.002

CrossRef Full Text | Google Scholar

Noroozi, O. (2018). Considering students’ epistemic beliefs to facilitate their argumentative discourse and attitudinal change with a digital dialogue game. Innovat. Educat. Teaching Int. 55, 357–365. doi: 10.1080/14703297.2016.1208112

CrossRef Full Text | Google Scholar

Noroozi, O., and Hatami, J. (2019). The effects of online peer feedback and epistemic beliefs on students’ argumentation-based learning. Innovat. Educat. Teaching Int. 56, 548–557. doi: 10.1080/14703297.2018.1431143

CrossRef Full Text | Google Scholar

Noroozi, O., Biemans, H., and Mulder, M. (2016). Relations between scripted online peer feedback processes and quality of written argumentative essay. Internet Higher Educ. 31, 20–31. doi: 10.1016/j.iheduc.2016.05.002

CrossRef Full Text | Google Scholar

Noroozi, O., Kirschner, P. A., Biemans, H. J., and Mulder, M. (2018). Promoting argumentation competence: Extending from first-to second-order scaffolding through adaptive fading. Educ. Psychol. Rev. 30, 153–176. doi: 10.1007/s10648-017-9400-z

CrossRef Full Text | Google Scholar

Paul, R. W. (1990). Critical thinking: What every person needs to survive in a rapidly changing world. Santa Rosa, CA: Foundation for Critical Thinking.

Google Scholar

Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educat. Psychol. Rev. 18, 315–341. doi: 10.1007/s10648-006-9029-9

CrossRef Full Text | Google Scholar

Pekrun, R., and Linnenbrink-Garcia, L. (2014). International handbook of emotions in education. New York, NY: Routledge.

Google Scholar

Pekrun, R., and Perry, R. P. (2014). “Control-value theory of achievement emotions,” in International handbook of emotions in education, eds R. Pekrun and L. Linnenbrink-Garcia (New York: Taylor & Francis), 120–141.

Google Scholar

Pekrun, R., and Stephens, E. J. (2012). “Academic emotions,” in APA educational psychology handbook, Vol. 2, eds K. R. Harris, S. Graham, T. Urdan, S. Graham, J. M. Royer, and M. Zeidner (Washington, DC: American Psychological Association), 3–31.

Google Scholar

Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., and Perry, R. P. (2011). Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ). Contempor. Educat. Psychol. 36, 36–48. doi: 10.1016/j.cedpsych.2010.10.002

CrossRef Full Text | Google Scholar

Pekrun, R., Goetz, T., Titz, W., and Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educat. Psychol. 37, 91–105. doi: 10.4324/9781410608628-4

CrossRef Full Text | Google Scholar

Pekrun, R., Vogl, E., Muis, K. R., and Sinatra, G. M. (2017). Measuring emotions during epistemic activities: The Epistemically-Related Emotion Scales. Cognit. Emot. 31, 1268–1276. doi: 10.1080/02699931.2016.1204989

PubMed Abstract | CrossRef Full Text | Google Scholar

Rosenberg, E. (1998). Levels of analysis and the organization of affect. Rev. General Psychol. 2, 247–270. doi: 10.1037/1089-2680.2.3.247

CrossRef Full Text | Google Scholar

Schommer-Aikins, M., and Hutter, R. (2002). Epistemological beliefs and thinking about everyday controversial issues. J. Psychol. 136, 5–20. doi: 10.1080/00223980209604134

PubMed Abstract | CrossRef Full Text | Google Scholar

Scriven, M., and Paul, R. (1996). Defining critical thinking: A draft statement for the National Council for Excellence in Critical Thinking. Tomales, CA: Foundation for Critical Thinking.

Google Scholar

Shuman, V., and Scherer, K. R. (2014). “Concepts and structure of emotions,” in International handbook of emotions in education, eds R. Pekrun and L. Linnenbrink-Garcia (New York: Routledge), 13–35.

Google Scholar

Siegel, H. (1988). Educating reason: Rationality, critical thinking, and education. New York, NY: Routledge.

Google Scholar

Simpson, E., and Courtney, M. D. (2002). Critical thinking in nursing education: Literature review. Int. J. Nurs. Practice 8, 89–98. doi: 10.1046/j.1440-172x.2002.00340.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Sinatra, G. M., and Broughton, S. H. (2011). Bridging reading comprehension and conceptual change in science education: The promise of refutation text. Reading Res. Quart. 46, 374–393. doi: 10.1002/rrq.005

CrossRef Full Text | Google Scholar

Strømsø, H. I., and Bråten, I. (2014). Students’ sourcing while reading and writing from multiple web documents. Nordic J. Digit. Literacy 2, 92–111.

Google Scholar

Strømsø, H. I., Bråten, I., and Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’ trustworthiness? Educat. Psychol. 31, 177–206. doi: 10.1080/01443410.2010.538039

CrossRef Full Text | Google Scholar

Tabachnick, B. G., and Fidell, L. S. (2013). Using multivariate statistics, 6th Edn. Boston: Pearson.

Google Scholar

Tiedens, L. Z., and Linton, S. (2001). Judgment under emotional certainty and uncertainty: The effects of specific emotions on information processing. J. Personal. Soc. Psychol. 81, 973–988. doi: 10.1037/0022-3514.81.6.973

PubMed Abstract | CrossRef Full Text | Google Scholar

Tippett, C. D. (2010). Refutation text in science education: A review of two decades of research. Int. J. Sci. Math. Educat. 8, 951–970. doi: 10.1007/s10763-010-9203-x

CrossRef Full Text | Google Scholar

Toulmin, S. E. (2003). The Uses of Argument. New York, NY: Cambridge University Press.

Google Scholar

Tremblay, K., Lalancette, D., and Roseveare, D. (2012). AHELO Feasibility Study Report. Volume 1 Design and Implementation. Paris: Organization for Economic Co-operation and development (OECD).

Google Scholar

Valero Haro, A., Noroozi, O., Biemans, H. J., and Mulder, M. (2019). The effects of an online learning environment with worked examples and peer feedback on students’ argumentative essay writing and domain-specific knowledge acquisition in the field of biotechnology. J. Biol. Educat. 53, 390–398. doi: 10.1080/00219266.2018.1472132

CrossRef Full Text | Google Scholar

Weiner, B. (2007). “Examining emotional diversity on the classroom: An attribution theorist considers the moral emotions,” in Emotion in education, eds P. A. Shutz and R. Pekrun (San Diego, CA: Academic Press), 73–88.

Google Scholar

Whitman, D. B. (2000). “Genetically modified foods: Harmful or helpful?,” in Canadian Standards Association: Discover Guides, (Ottawa, ON: Government of Canada).

Google Scholar

Yang, F., and Tsai, C. (2010). Reasoning about science-related uncertain issues and epistemological perspectives among children. Instruct. Sci. Int. J. Learning Sci. 38, 325–354. doi: 10.1007/s11251-008-9084-3

CrossRef Full Text | Google Scholar

Zeidner, M. (2014). “Anxiety in education,” in International handbook of emotions in education, eds R. Pekrun and L. Linnenbrink-Garcia (New York: Routledge), 265–288.

Google Scholar