Engineering students’ justifications for their selections in structured learning diaries

Introduction: New digital tools such as structured learning diaries (SLD) can serve as both a measurement and intervention tool, at scale. However, we do not yet know how students use the SLD tool and justify their actions within it. Clickable items are easy to use, but do these diaries hold any deeper meaning for students? This study aims to explore the means of justification used by students to create SLD content, their profiles based on these justifications, their levels of reflection, and the relationship between the profiles and use of SLDs.

Methods: We interviewed a sample of first-year master’s-level engineering students to gather justifications and reflections related to the content of their SLDs. Rank- and median-based statistical tests were used to explore the connections between the interview-based profiles, and diary behavior was analyzed through log data.

Results and Discussion: Our findings revealed distinctive profiles with different characteristics related to the structured learning diary behavior, including differences in how changes of difficulty and emotion ratings were made in SLDs. This study opens up a new area for future research and encourages the development of structured learning diary tools as a means of monitoring changes in student thinking at scale.

1. Introduction

The study of cognitive psychology has been ongoing since the 1960s, with a particular focus on mental models and mental processes of knowing. Epistemology, the study of the nature of knowledge and knowing, has its roots in philosophy dating back to Plato. Within educational research, there has been a growing interest in mental processes of knowing since Perry’s (1970) developmental research on college students and Kitchener’s (1983) multidimensional model. This research has been used to realign epistemological psychology with philosophy, focusing on how knowledge and knowing are justified.

One line of research in epistemic psychology, informed by philosophy, has centered on justifying knowledge and knowing. For example, Muis (2008) investigated undergraduate students who differed on how they justified knowledge, categorizing them into rationalists, empiricists, and those who are both rational and empirical. These profiles were then used to find relations to self-regulated learning and mathematical problem-solving. The present study aims to align with philosophy-informed epistemic research by adapting the framework to the context of structured learning diaries.

Despite the growing interest in epistemic cognition and the learning process, most studies have focused on students’ beliefs rather than cognitive processes. In particular, researchers have been urged to use log traces to capture processes instead of relying only on self-reports. One possible solution has been the use of digital learning diaries, which allow researchers to record students’ states and behavior during the learning process. However, little research has been done to investigate the epistemological dimensions related to the use of diaries, despite the acknowledged capacity for tracing epistemic cognition afforded by the development of computational methods.

Studies on epistemic beliefs often use interviews and questionnaires, i.e., general, or domain/topic-specific Likert-scale self-reports and surveys (e.g., Royce, 1967; Bråten et al., 2005; Hofer and Sinatra, 2010; Bråten et al., 2019; Lonka et al., 2021). However, the measurement procedures themselves can evoke epistemic challenges for participants; in understanding the questionnaire items, limitations in expressing their thoughts, and personal capabilities reporting on their metacognition (Barzilai and Weinstock, 2015). In addition, recent research suggests that epistemic cognition is not stable across different contexts and tasks (Buehl et al., 2002; Hofer, 2006; Muis et al., 2006; Bråten et al., 2008; Barzilai and Weinstock, 2015). Instead, Sandoval et al. (2016) call for researchers to develop, test, and refine measurement instruments to meet epistemic ideals for psychometric validity; the “right tool for the right job.” They suggest objective observations as a critical component in the measurement of epistemic cognition. The present study addresses these methodological challenges with mixed-method research, as suggested by, e.g., Greene and Seung (2014).

There is a growing need (e.g., Eastwood et al., 2017) to advance methodological approaches and to understand how epistemic cognition is related to behavior in digital environments. New data analysis techniques can add a “new dimension through which to understand the online learners” and “add a new variable with which to structure and predict mastery training” (Johanes, 2017). This study attempts to understand how justification patterns are related to students’ reflective journaling in a digital learning tool (the structured learning diary). When we better understand how students with different justification profiles interact with digital tools, such as a structured learning diary, we can take the results back to practice and design tools for larger audiences.

This study aims to investigate how students justify and reflect their own learning experience within a digital learning tool, i.e., a structured learning diary. Our study seeks to address the gap in the literature by investigating whether structured learning diaries could be used as a tool to monitor and regulate students’ learning, which could lead to improved learning outcomes in academic and professional settings.

2. Theoretical and empirical background

2.1. Justification of knowledge as a part of epistemic cognition

Epistemic cognition refers to the ways individuals think about knowledge and how it is acquired, evaluated, and justified, reflecting on the limits of one’s knowing, the certainty of knowing, and the criteria of knowing (Kitchener, 1983; Greene et al., 2016; Hofer, 2016; Greene et al., 2018). Epistemic cognition has been argued to be a key to higher-level cognitive outcomes and critical thinking skills (Sandoval et al., 2016), both of which are highly valued in higher education curricula. Epistemic cognition plays a crucial role in learning and academic achievement.

In this study, we focus on justification of knowledge and knowing. This dimension of epistemic cognition comes from the groundwork laid by Hofer and Pintrich (1997) and their model of multidimensional beliefs, which includes the simplicity of knowledge, the certainty of knowledge, the source of knowledge, and the justification for knowing. Their definition of personal epistemology as the nature of knowledge and knowing, not the nature of learning, has since been challenged. Elby (2009) argued that the definition should not be defined a priori before the empiric evidence. Moreover, Greene et al. (2008) worked to realign educational epistemic cognition research with philosophy. They argued that only the justification for knowing is truly epistemic and that this dimension should be expanded to differentiate between personal justification and justification by authority. Further, empirical findings by Bråten, Ferguson and their colleagues (Ferguson et al., 2012; Ferguson and Bråten, 2013; Bråten et al., 2014, 2019) resulted in expanding the justification for knowing with a third dimension: justification by multiple sources.

2.2. The three dimensions of justification of knowledge

Personal justification is a knowledge claim based on an individual’s own knowledge. Evaluating knowledge claims through expert sources refers to justification by authority. Justification by multiple sources is cross-checking and comparisons between several sources of information (Bromme et al., 2010a; Bråten et al., 2014, 2019; Kammerer et al., 2021).

These three means of justification have been studied in several contexts. Bråten et al. (2013) found that personal justification of knowledge was a negative predictor and that using multiple sources for justification had a positive effect on multiple-text comprehension. In a later study (Bråten et al., 2014), justification by multiple sources was identified as a positive predictor of topic-specific multiple-text comprehension when controlled with prior knowledge and the other justification means (i.e., personal justification, justification by authority). Mason and colleagues (Mason et al., 2010, 2011; Mason and Bromme, 2010) who studied web-based learning using think-aloud protocols found that those who used science-based justification performed better than those who reflected on their knowledge claims through personal justification.

Another central line of research focuses on internet-specific epistemic cognition. For example, in their study using the Internet-Specific Epistemological Questionnaire, ISEQ (Bråten et al., 2005), Kammerer et al. (2015) found that students who believed that Internet-based information needs to be compared across multiple sources tended to use more reliable sources on the Internet. Based on their findings related to the three justification dimensions, Bråten and colleagues developed and validated a new instrument: The Internet-Specific Epistemic Justification Inventory (ISEJ) (Bråten et al., 2019). Kammerer et al. (2021) used this new instrument to study the relationship between epistemic beliefs and behavior during a 20-min web search task. They found that a higher measure of justification by multiple sources predicted participants’ post-test written justifications, their length, and the number of relevant content items. They found that this justification dimension did not predict, however, the extent of queries, web-page navigation, the opening of multiple browser tabs, or think-aloud comments.

Ferguson and Bråten (2013) suggest, based on several reviewed studies, that epistemic beliefs are a relatively independent individual difference variable, “making it reasonable to [--] profile students [--] and examine emergent profiles in relation to other, external variables.”

We employ our adaptation of the justification of knowledge framework to investigate how students justify and reflect their actions within a digital learning tool. Personal journaling, such as structured learning diary content and students’ reflection of same, is not epistemic as such. Rather, we see that as a derivate of one’s epistemology. Nevertheless, we need to understand the meaning of a student’s entries in their personal journaling tool. Adapting the justification of knowledge into justification of actions in the context of a structured learning diary, builds new insights to assess such digital tools. In addition, the adaptation adds comparability, and can reveal new perspectives that have not been observed in the original contexts. The knowledge items in this study, i.e., the diary entries, are highly personal. Thus, it is reasonable to argue that such knowledge would be predominantly personally justified. However, to our knowledge, this has not yet been studied.

2.3. Structured learning diaries

Reflection on learning content or learning behavior is the general goal of learning diaries (Fabriz et al., 2014). Structured learning diaries (SLDs) work in a similar way, achieving this through structured/pre-defined objectives, different types of standardized question items, such as Likert items, open questions, or nonverbal pictures or emojis, answered repeatedly (Schmitz and Wiese, 2006; Klug et al., 2011; Kivimäki et al., 2019).

SLDs have drawn special interest in educational science research concerning self-regulated learning as an intervention tool (Schmitz and Perels, 2011; Fabriz et al., 2014; Ferreira et al., 2014; Panadero et al., 2016; Broadbent et al., 2020; Pesonen et al., 2020) and capturing sequences of states for situational self-regulation (Schmitz and Wiese, 2006; Geiser et al., 2017). Despite the wide interest within the research community in enhancing self-regulated learning with tools like the SLDs, little has been done to find out what students think when they are using these tools.

Learning diaries are usually implemented only for a few weeks (Fabriz et al., 2014). Fabriz et al. (2014) suggest using diaries for longer, even a full term, to record “the development of a beginning learner to the point of exam preparations.” Considering that SLDs can require significant effort from students (Klug et al., 2011; Pesonen et al., 2020), the present study explored the use patterns of diaries over a time of one academic year, followed by a retrospective interview at the end of the term.

Digitalized SLDs can record rich data related to the course content and learning behavior. The diary used in the present study has been designed to record students’ activities and reflections on their learning, guiding the student to think of questions such as “How well do I think I learned this course topic?,” “How difficult was this for me?,” and “Am I excited about this or rather bored?” This part of the reflection process aims to foster students’ self-monitoring activities and, at best, result in new decisions and actions, i.e., turning reflection into reflexivity.

The present study explores students’ judgments of their reflections: how did students end up making their diary selections? However, the judgment strategies used by students do not indicate how far along (or open) the students are in the process of turning their self-monitoring into actionable changes. Thus, the present study adapted another framework, the 3R-EC framework, to investigate where students stand in the terms of their reflection.

2.4. From reflection towards reflexivity

For a deeper understanding of how students justify their diary entries, we adapted another coding scheme to measure the reflexivity of the participants’ reflections. Here we present a framework that fits well with the context of the present study. SLDs are tools for recording student reflections that can lead to behavioral or conceptual changes through, e.g., self-regulation or epistemic cognition. The model presented next was originally used in the context of teacher training.

In the context of teachers’ personal epistemologies, Schraw et al. (2017) have argued towards a process-oriented approach in understanding personal epistemology. In their 3R-EC framework, reflection proceeds towards decision-making, i.e., reflexivity. In other words, people use reflexivity to justify the cognitive processes that follow, by thinking about the knowledge itself and questioning and refining epistemic cognition through reflection (Marcus, 2021).

In the present study, the 3R-EC model can give us information on the state of students’ thinking. One of the critical purposes of learning diaries is to make students’ reflections explicit. At best, this reflection leads to more informed actions. An adapted version of the 3R-EC model can be used to assess our participants’ stage in their reflection/reflexivity process (Figure 1).

FIGURE 1

Figure 1. The 3R-EC framework. Adapted and re-contextualized from Brownlee et al. (2017), Lunn Brownlee et al. (2019), Schraw et al. (2017), and Brownlee et al. (2016).

Earlier studies with this model have focused on teacher education (Lunn Brownlee et al., 2017, 2019). Teachers are seen as experts that undergo personal growth processes related to their teacher identity. In our context, the students are master’s-level students who focus on building their competence in multiple course contents. The primary purpose of the interviews was to make students’ reflections in the learning diary tool explicit. This framework was assumed to reveal students’ readiness to act based on their reflections in our study. This change in epistemic cognition, as Lunn Brownlee et al. (2017) suggest, “may take place through reflexivity, not just reflection [--].”

The framework serves two needs in the current study. The (epistemic) change is at the core of the goals of the learning diaries. Therefore, it is relevant to explore the participants’ reflection, i.e., do students differ from each other in an interview setting and how do the possible differences align with the participants’ justification strategy use. In addition to the explorative purpose, the framework serves as a method to control that the participants’ use of justification strategies is not due to individual differences in the generic ability to reflect in an interview situation.

2.5. Present study

The purpose of this study is to delve into the relationship between students’ justification of their behaviors in digital learning tools. The literature on epistemic cognition and structured diaries has motivated us to investigate the ways in which students justify their selections in their diaries, and whether this variation is related to their actual behavior in the diary tool. Our focus is on master’s-level students and their cognitive processes when justifying the content of their learning diaries on given course topics. We are interested in understanding, e.g., why they selected a competence evaluation rating of “4” (on the scale of 1, low, to 5, high), for each topic, how confident they are in their selections, and whether they would like to make any changes to their selections afterward. Considering the literature, we have formulated research questions and hypotheses.

2.5.1. Research question 1: how do students justify their selections in their structured learning diary?

Each participant was asked to justify their selections in SLDs in a retrospective, stimulated recall interview. We then analyzed the answers using a coding scheme built on the three means of justification of knowledge dimensions.

Hypothesis 1: Literature suggests that students would justify through personal justification, justification by authority and justification by multiple sources in several contexts (Bråten et al., 2013; Ferguson and Bråten, 2013; Kammerer et al., 2021). In this new context, personal journaling, we expect that students would mainly express a personal justification strategy when asked to justify their selections in their SLDs.

2.5.2. Research question 2: can we identify profiles based on the use of different justification means and level of reflection?

We formed profiles based on the participants’ justification strategies in their interviews. Characteristics of the profiles were further explored based on the reflectivity of the participants (3R-EC framework).

Hypothesis 2: Students would employ multiple justification strategies, thus making it possible to identify profiles (Ferguson and Bråten, 2013). Also, we expected students to differ in how they express reflection versus reflexivity in their interviews (Kammerer et al., 2015).

2.5.3. Research question 3: are justification profiles related to structured learning diary use?

We investigated whether the students’ justification means (profiles) relate to actual diary use (log data). When students started using the diary tool, they were told that they could also update their diary and, e.g., change the numerical items. We assume that, in particular, making changes in the diary would indicate traces related to epistemic cognition and students’ overall tendency to reflect on their learning.

Hypothesis 3: Based on the finding of Kammerer, Bråten and their colleagues (Bråten et al., 2005; Kammerer et al., 2021), we expected to find differences between the means of justification and use of SLD. Multiple sources was expected to connect with an active reassessment in the learning diary tool: for instance, at first making a selection on a structured diary item and later changing that original selection as a result of a reflective process. The rationale behind this hypothesis is that students who are more open to changing their justification by external sources (authority and multiple sources) would also respond with more dynamic SLD behavior.

3. Method

We employed a mixed-method approach to answer the research questions, considering the explorative aims of the present study. When compared with the traditional dichotomy between qualitative and quantitative methods, mixed-methods approaches can build a more comprehensive picture of the studied phenomena (Sammons, 2010; Teddlie and Sammons, 2010). This power comes from integrating, e.g., several data collection or analysis methods to corroborate the findings (Johnson et al., 2007).

Data was collected through interviews, and log data from the SLDs. This technique where the phenomenon is measured and analyzed in different times, people, or settings (Cohen et al., 2018) can increase the researcher’s confidence as the inferences can be backed by several data points and methods. Thus, the present study employed mixed-method (Tashakkori and Teddlie, 2016) research characteristics throughout the study, as described in Table 1.

TABLE 1

Table 1. Mixed-method characteristics in the present research design and its goal.

3.1. Participants and procedure

The current study was conducted at a large research university in Finland in a master’s program in mechanical engineering (2 years / 120 ECTS). The program is the participating university’s largest master’s program by student intake. The students had finished their bachelor’s degree prior to starting their studies in this master’s program.

The targeted study population was the test group of the learning diary tool in a randomized controlled trial (N = 64). We invited a computationally randomized (Urbaniak and Plous, 2013) sample from the test group to participate in the interviews for the present study. The interviews were conducted after the trial had ended and all students (both test and control group) had used SLD.

We controlled the invited students based on their diary activity. An individual SLD activity level was calculated for each student based on trace/log data by calculating the number of changes for each SLD feature during the first 14 weeks (fall semester). The activity was scaled to an interval [0, 1], and the average was calculated for every student. Based on this measurement, the students were divided evenly into high, medium, and low activity users. Each group consisted of 21 students who were participating in our research and had submitted their SLDs. A sample of ten students from each activity-based group was randomly selected to be invited for an interview. Twenty-eight students out of the sample size of 30 agreed to be interviewed (93%). Two of the participants took part in developing the interview protocol and were excluded from the final analysis. Thus, the final sample was 26 students (22 male, 4 female). The mean age was 26.85 years (SD = 2.64).

The use of the SLD was a mandatory task for the students. Participation in the study was optional. One student decided to opt out from this study. All participants gave informed consent to take part in this study.

3.2. Data and materials

3.2.1. Interview

Interviews were carried out with two interviewers. First, we piloted the initial interview questions with two students who were not part of the sample. Then, a student in the final sample was interviewed with both interviewers to test the interview protocol further and to calibrate for future interviews. After this, the interviewees were divided evenly between the two interviewers. Midway through the interviewing process, the interviewers reflected on their interview experiences. No significant changes were made based on this reflection.

All interviews were conducted during a period of 5 weeks at the end of the academic year. One student asked to be interviewed several months later. This request was denied to maintain the temporal consistency of the study. In one interview, we failed to record audio. This interview was excluded from the analysis. In total, 26 interviews were successfully recorded and transcribed into English, with the following distribution related to their SLD activity: high activity = 8 participants, medium activity = 9 participants, low activity = 9 participants. Interview durations ranged from 17 to 52 min, averaging 31 min (Mdn = 30, SD = 9).

During the interviews, we employed the stimulated recall method (Bloom, 1953), using each participant’s learning diary as a cue for memory retrieval. We provided sufficient time for participants to review their diary content on a laptop computer and even allowed them to make updates during the interview. To gain insights into students’ cognitive processes, we asked participants to justify their selections while simultaneously allowing students to browse their diaries to support their answers (Sime, 2006; Mackey and Gass, 2011). Our focus was not on investigating the epistemology related to the course topic or content but rather on the journaling content itself. We explored how students justify their selections, such as whether they rely solely on personal thoughts or on authoritative sources.

At the beginning of the interview, the interviewer opened the student’s SLD that they had submitted at the end of the first semester. The computer was set to record the screen and audio. The student was handed a computer mouse, and they were able to fully interact with their diary content and view. Next, the interviewer asked the student to look at all their selections and updates and to comment on them: why did you make this selection, how sure are you that your self-assessment is correct, and has your assessment changed after that semester, and if it has, how? The interviewer also asked the student to recall what they learned about that topic and what they did not learn.

3.2.2. Structured learning diary

We used a mind map-based, digital SLD tool designed by Kivimäki et al. (2019). The SLD tool has a curriculum-based mind-map structure that extends from the degree program node to the major/minor subject level nodes and further into the course and course topic nodes. Students were instructed to reflect on their competence, experience of difficulty, and feelings; to write notes; and to draw arrows to describe relations between nodes (see Table 2 and Figure 2).

TABLE 2

Table 2. Descriptions of the SLD use (log data) metrics.

FIGURE 2

Figure 2. Excerpt from a learning diary showing all main diary elements: 1) Course code and name (in template), 2) Course topic (in template), 3) Competence (blue background; selectable scale from 1, low, to 5, high), 4) Emotion (student selects between emoticons: excited, relaxed, neutral, bored and anxious), 5) Difficulty (black background; selectable scale from 1, easy, to 5 hard), 6) Relation arrow (Drawn by a student from, e.g., topic to topic or course to topic), 7) Comment (added by student).

Students were instructed to submit their most recent file weekly through a learning management system for 14 weeks, even if there were no changes. Students rated their assessments of their competence, perceived difficulty, and current emotions related to the course topic. The topics were derived from the curriculum and revised by the instructing teachers. A typical course consisted of five to ten topics. The students were instructed to use relation arrows to visualize connections they perceived between courses and topics. They were also instructed to name those connections. Students were instructed to use notes freely, in any way they considered meaningful. Students were not given any personal feedback on their diaries. However, the data in the diaries were aggregated on a group level and visualized on a public dashboard. All students interviewed changed their competence, difficulty, and emotion selections; 19 students drew relation arrows; 21 wrote comments.

The files and their contents constitute several log data dimensions. In this study, we are interested in measuring the use and changes made in SLD that we expect to reveal traces related to epistemic cognition and reflection.

3.3. Interview coding schemes and reliability

3.3.1. Interview coding

We used the three-dimensional framework of the justification of knowledge by Ferguson and Bråten (2013) to classify the content of the interviews (see Table 3). The framework was adapted into our context of personal journaling as justification of actions and competence in SLD. Justification by authority can refer to the evaluation of external sources. In the context of students’ justification of personal diary data, our coding scheme investigates whether the student justifies, e.g., high competence on the topic based on the course grade. In multiple sources, a student can, for example, justify reflection based on several points of view (sources) that the student combines to construct the justification claim.

TABLE 3

Table 3. Operationalization of the three justification dimensions into the context of SLD use.

Quotations used in the three-dimensional framework of justification were further coded based on the 3R-EC coding scheme presented in Table 4. Those parts of interviews where the participant was clearly reflecting were coded based on this scale from one to four (1 = reflective commenting, 2 = comment on efficacy, 3 = internal dialog, 4 = reflexive planning), respectively. Due to the process-oriented nature of this scale, each participant’s highest score was defined as a representation of their stage in the process of reacting to their reflection. In other words, if the student is only using reflective commenting, they were assigned the value one. Value two was assigned to a student who reflects on their efficacy (in addition to commenting). Students who undergo an internal dialog while answering the questions in the interview are inferred to be further in the process of reflection, thus assigned with value three. Students who go further towards planning actionable reactions based on their reflection are scored with a value of four.

TABLE 4

Table 4. Operationalization of the 3R-EC framework into the context of SLD use.

3.3.2. Inter-rater reliability of coding schemes

Inter-rater reliability between the two raters was measured with a randomly selected 20% subset of 301 coded interview quotations (see Table 5). We used a rater with a master’s degree in educational sciences who has not participated in the analysis nor is an author of this study. The reliability was assessed by using Cohen’s Kappa. We used unweighted kappa for the three means of justification, as this is categorical data. Therefore, four levels of reflection and reflexive thinking are parts of a sequential process and can be seen as interval-type data. In the latter, we used weighted kappa with equal intervals.

TABLE 5

Table 5. Interrater reliability for the used interview coding schemes.

4. Results

4.1. Students’ use of justification strategies in the interview

Interview data were used to create profiles based on the three means of justification. Personal justification was used by all participants, when they were asked to justify their actions in SLD, e.g., the assessment of their competence on the topics in the diary. When a student was using personal justification, they referred only to their own judgements, e.g., “computer-aided tools, well, I did well with those” or “I gave that a four, so I’m pretty knowledgeable about it.” Whereas some students also found justification for their competence assessment from authority sources, mostly course grades “there’s a long line of fours [in the diary], which does correspond to the grade I got from the course” but also discussions with authority sources like teachers, professors, or academic advisors. Some justified their assessment based on course requirements and even experience in job interviews. Justification by multiple sources presented a variety of sources, such as prior experience of working in the field “living in India … and worked enough to have a clear understanding,” comparisons with peers “it was much the other ones who did that … that’s why I gave it a two, because I wasn’t able to participate in it, or I did not participate that much,” comparisons to experiences in prior courses “I did not grade these as highly at first, but towards the end, I grew more confident about that stuff. These things have been individually discussed quite a lot on other courses. It’s only the two lowermost ones that have not been practiced a lot.” Several students compared their own competence based on external sources: “so it’s a pretty simple feature. You can find good instructions on the internet… very simple, but I cannot do it perfectly, so therefore there’s a four.”

Each justification strategy was used as a threshold for a profile. The criteria and participant distribution are presented in Table 6.

TABLE 6

Table 6. Distribution of justification strategies used by students and criteria for each profile.

Based on the students’ use of justification strategies, the following profiles were formulated and labeled as: Students using personal justification (PJ), students using authority justification (AJ), students using multiple sources justification (MS), and students using different justification strategies (DS).

The interviewed students had been randomly selected based on their general activity of using the diary tool. Based on the profiling on justification means they were distributed into the profiles as follows: PJ (high activity = 1, low activity = 2), AJ (medium activity = 3, low activity = 2), MS (high activity = 4, medium activity = 3), DS (high activity = 3, medium activity = 3, low activity = 5). All profiles represented both more and less active users of SLD. Activity was not used as a variable in defining the profiles. Moreover, the distribution of any activity levels did not seem to accumulate on any certain profiles.

4.2. Justification profiles and type of reflection

Some students reflected in the interview only at a vague level where they as subjects (“I guess,” “I thought,” “it was nice to spot”) commented objects (“those things,” “product blueprints”). Characteristic to this subject-object reflection were comments starting with, e.g., “I was just thinking.” More students reached a level of reflection where they combined efficacy into their reflection, such as “…so you know how to actually do it,” “I have thought about the field and what it requires from me and how I’d manage in it and how competent I’d be.” Most students reached internal dialog, where they made observations about some objects and built new thoughts based on that, such as“Actually the knowledge was quite a lot but the thing is that mostly I knew that to study by yourself because going to the lectures, I’m still do not [sic!] get the point of it because what we were studying in the lectures has nothing to do with what the task at home from the assignments was. So, a little bit frustrating but that’s fine, still I liked it.” Two students built on their dialog to the lengths of reaching reflexive planning “so I wonder if there is a more professional way to go about it,” “I made an Excel study plan of my own which helped me to follow and plan my credits and stuff,” “And if the idea is not fully formed, so you have like a faint idea in your head … but you can think and try to come up with a solution.”

The highest stage of the reflexivity process reached by the students during the interview was used to add another dimension to the justification profiles. Table 7 shows how students’ reflectiveness falls into the justification profiles.

TABLE 7

Table 7. Distribution of students with varying reflection/reflexivity by justification profiles.

PJ students’ reflection was rather deep (i.e., reaching the level of internal dialog during the interview). AJ students displayed various stages of reflection/reflexivity. Also, MS students were divided between students using internal dialog and students reflecting on objects with or without efficacy notions. The DS students were the most reflective in the 3R-EC scale. This group steadily used internal dialog (n = 9), and two students employed reflexive planning (n = 2) in their interviews.

4.3. Justification profiles and structured learning diary use

The log data were aggregated on a group level for each justification profile. Based on descriptives, the students who used multiple sources justification (MS) seemed to be the most active in making changes inside their diaries, i.e., selecting and then returning to change their original selection during the semester, apart from the number of comments (highest among students who used different justification strategies; see Table 8).

TABLE 8

Table 8. Diary log data on group level metrics.

Since the groups were small and the scales not normally distributed, we performed Kruskal-Wallis non-parametric tests for all the presented log data metrics for AJ, MS, and DS students. PJ was dropped from analysis due to low group size (n < 5). Only the number of difficulty selection changes across the justification groups provided evidence of a statistically significant difference (p = 0.016) between the mean ranks of at least one pair of groups (Figure 3). Three different pairings were further assessed with Mann–Whitney tests with a Bonferroni adjusted alpha level of 0.016 per test (0.05/3). In MS, difficulty changes (Mdn = 73) were done more often than in AJ (Mdn = 43; U (NMS = 7, NAJ = 5,) = 2.00, z = −2.522, p = 0.009). Likewise, in MS, difficulty changes were (Mdn = 73) more likely than in DS (Mdn = 48); (U (NMS = 7, NDS = 11,) = 10,50, z = −2,538, p = 0.009). We did not find statistically significant differences between the ranks in AJ and DS after Bonferroni adjustment, (U (NAJ = 5, NDS = 11,) = 23, 50, z = −0.454, p = 0.680).

FIGURE 3

Figure 3. Difficulty changes by justification profiles (Kruskal-Wallis test).

Other log data differences between groups were not found to be statistically significant when subjected to the Kruskal-Wallis test. However, since the emotion change differences between profiles had similar median changes as those in the difficulty variable, we performed compare means tests for both difficulty changes and emotion changes. The independent-samples median test with a grand median of 47 suggested differences between some of the justification groups in both difficulty changes and in emotion changes, p = 0.010 and p = 0.046, respectively. Based on this finding we decided to run a pairwise Mann–Whitney U test for AJ, MS, and DS in the emotion changes variable as well. In MS, emotion changes (Mdn = 69) were done more often than in AJ (Mdn = 42; U (NMS = 7, NAJ = 5) = 1.50, z = −2.603, p = 0.008). No statistically significant differences were found between AJ (Mdn = 42) and DS (Mdn = 42) or MS (Mdn = 69) and DS (Mdn = 42) related to the rank based on the number of emotion changes: U (NAJ = 5, NDS = 11) = 27.00, z = −0.057, p = 0.977; U (NMS = 7, NDS = 11) = 19.00, z = −1.766, p = 0.085; respectively (see Figure 4).

FIGURE 4

Figure 4. Emotion changes by justification profiles (Median test).

5. Discussion

This study revealed that university students justify their diary use with a variation of strategies and that most students have made meaningful choices in their structured learning diaries.

In the interviews, we investigated whether master’s-level students would use all three means of justification of knowledge dimensions when justifying their learning diary content. Based on the results, we identified four groups with differing strategy use and level of reflection. We then tested how the grouping relates to learning diary use. Based on the statistical non-parametric tests, we found one group, namely students using the multiple resources as justification strategy (MS), that differed from the others in terms of revisiting and changing their difficulty estimates on course topics more often. Thus, an association was found between students’ justification strategies and the level of reflection (self-presented via interviews) and objective behavioral data of learning diary use. Next, we present the results related to the research questions and hypotheses.Finally, we discuss the results further, elaborate on limitations, and present future directions.

5.1. Hypothesis 1: students express three means of justification

We identified all three justification strategies in our interview data: personal justification, justification by authority, and justification by multiple sources, as suggested by the literature (Ferguson et al., 2012; Bråten et al., 2013; Ferguson and Bråten, 2013; Kammerer et al., 2021). Our results showed that all participants used a personal justification strategy, which was expected considering the personal journaling content of the task. However, only three students (PJ) used this strategy exclusively when justifying their selections in the SLD tool.

Our findings suggest that justification of knowledge and knowing, reflecting a part of students’ epistemic cognition, is applied in at least four different ways in the SLD context. It is not surprising that all students in our sample employed largely personal justification due to the self-reflective nature of the content, a personal learning diary, and the protocol used to collect the data, an individual interview. Further, we distinguished participants based on whether they used justification by authority (AJ) or justification by multiple sources in addition to the personal justification dimension. We identified two groups who used multiple sources: Students who mainly used multiple sources justification (MS) and students who alternated between a variety of justification strategies: personal justification, justification by authority and justification by multiple sources (DS).

5.2. Hypothesis 2: distinctive justification profiles can be identified

Students used distinctive means for justifying the action statements in their interviews. Based on these varying means of justification, we were able to recognize four profiles: PJ, AJ, MS, and DS. To further examine these profiles, we used the 3R-EC framework to find out whether some of the interviewed students employed deeper reflection than others. We expected the interviewed students to be at different stages of the reflectivity/reflexivity process, or at least vary in the amount of reflection (Kammerer et al., 2021). This was partly confirmed (Table 8). However, results showed that especially students who used multiple sources (MS and DS) displayed different stages of reflection/reflexivity in their interviews. Several MS students reflected only by commenting on the object of discussion and reflecting on their efficacy. In contrast, all DS students reached internal dialog and two reached reflexivity, which suggests that these students are able and willing to reflect at an advanced level and reach conclusive resolutions (reflexivity) based on their internal dialog. The findings related to reflection/reflexivity are, however, descriptive in nature.

5.3. Hypothesis 3: patterns in justification use relate to diary use

Earlier empirical studies have demonstrated a close relationship between the self-reported use of metacognitive strategies like monitoring and more sophisticated epistemic cognition (Bromme et al., 2010b). The SLD offers several means for monitoring strategies, e.g., monitoring one’s competence on topics, perceived difficulty, and experienced emotions (Kivimäki et al., 2019; Kivimaki et al., 2021). We found that MS students differ from AJ and DS student profiles in their diary use. They were the most active in changing their difficulty assessments on course topics during the semester. In addition, MS students were more active in changing their emotion ratings than AJ students.

In their study on the role of justification beliefs in naturalistic Internet searches by university students, Kammerer et al. (2021) found that the multiple sources strategy was positively connected with the extent and relevance of written justifications given after the search. Nevertheless, they found that multiple sources did not predict query and navigation activities, i.e., the number of websites visited or revisited. Our results are dichotomous. We found a connection between MS students and the number of visits/revisits to change structured item selections in their diaries (difficulty), whereas with DS students, we did not. In our study, we had two types of students who justified by multiple sources: those who were shifting between all three justification strategies in different topics (DS), and those who were consistent in mainly using multiple sources strategy across all topics and who also visited/revisited their difficulty reports in diaries more than any other group (MS).

5.4. General discussion, limitations, and future directions

We found that the MS group, or the students who mainly used multiple sources justification in addition to personal justification, was the most active in using the structured clickable items in the learning diary, returning to change them many times afterward, indicating self-evaluation, reflection, and use of metacognition in their learning processes more often than the others. On the other hand, DS students (students using multiple different justification strategies, including justification by multiple sources) executed deep reflection in their interview yet were not among the most active in their structured diary use. Kammerer et al. (2021) found in their study that beliefs in justification by multiple sources play a role in creating a more comprehensive representation in tasks based on Internet searches. That was in line with their theory-based assumptions. Still, contrary to the prior research, they (Kammerer et al., 2021) found that beliefs in justification by multiple sources did not predict how much participants searched or navigated websites, nor did it predict their comments on source evaluation and corroboration. Similarly, we found that those students who justify by multiple sources are divided into two different groups, rather than forming a single concise group.

Justification groups were linked with difficulty and emotion changes in the SLD. Interestingly, the link between the justification groups and competence ratings in the SLD was non-existent or weak at best. One explanation could be that students perceive competence self-assessment as tightly linked with a formal assessment (course grade). Several students mentioned this in the interview. Assessing one’s own difficulty experience, on the other hand, could be perceived as a more personal measurement that is not formally measured by any other measure. The difficulty self-assessment dimension could therefore be reflecting metacognitive activities rather than the competence measure. Hence, in addition to competence, difficulty and emotion assessment should be considered an area of interest in learning analytics applications to record students’ metacognition.

The present study explored connections between students’ justification strategies and the use of a learning analytics tool. The results show promising links between justification strategies and reflective behavior in a digital journaling tool. In our sample, it were the difficulty and emotion items that resonated most with the justification profiles. One conclusion could be that perhaps some students find at least difficulty ratings as a scale where justification by multiple sources is triggered more often. Researchers should study these connections further, e.g., for their effects on achievement.

PJ students were able to employ internal dialog in their interviews despite having their justifications only on a personal level. Perhaps these students were not experienced in justifying their views, or they held a clear belief that diary content is solely personal by nature. AJ students were a diverse group of students in terms of reflection. They, one-fifth of the interviewed students, based their justifications on a personal level, and on authority sources. One explanation here could be the engineering-domain specificity, where functionality and relying (and believing) on empiric tests is at the core of the nature of knowledge (McComas and Burgin, 2020).

The sample of students in this study (N = 26) was somewhat representative due to the random selection of participants and our efforts to ensure that we had students of each learning diary activity group. However, The PJ group consisted of only three students, limiting any further analyses and inferences from this profile. Further, due to the focus on qualitative data and analyses, the statistical tests should be considered as suggestive rather than focusing on significance due to the small sample size in these analyses. The interview coding resulted in a strong level of agreement between two independent raters. The metrics related to the students’ learning diaries were calculated computationally and quality checked by one of the authors. Despite the limitation in the sample size, the rich data from multiple perspectives (interviews and objective diary log data) increases the diversity of this study’s findings; the main finding being the connection between means of justification and behavior when using a digital SLD tool. Although we followed a stimulated recall protocol, retrospective interviews inevitably affect the inferences. We cannot draw conclusions on what happened at the time the student originally made the real-time selections.

In the present study, we combined the procedural reflectivity/reflexivity of the 3R-EC framework, with the three-dimensional justification model. This should be an area of interest for future studies, as Lunn Brownlee et al. (2017) assumed that epistemic cognition changes through reflexivity. If we could trace students’ reflexivity, we could design more informed prompts to guide and support students in their process of turning reflection into actionable decisions, e.g., supporting students’ self-regulation. Another area of interest is the number of comments. In our sample, these varied between 0 and 45. The median number of comments suggested differences between profiles. However, due to high variation within profiles, statistical significance was not reached. Other studies have also reported high variation in the number of students’ verbal comments during a web search (Bråten et al., 2014; Kammerer et al., 2021). An interesting question arises: why is there such a variation and should digital tools or instruction consider intervening here?

In sum, students differed in whether they employed one or several justification strategies when reflecting their actions in their structured learning diaries. Based on this, we recognized four student profiles. We found that the students who used justification by multiple sources could be divided further into two profiles using justification by multiple sources, either solely using this approach or alternating with other strategies. These two groups also seemed to differ in their diary tool use, the former showing more activity. Our study suggests that the frequency of changes in students’ self-assessments on a course topic’s difficulty and their emotions are linked to the use of justification by multiple sources. This could mean that students general justification capabilities relate to structured learning diary use. In turn, the use data of such diary tools could inform about the changes in student’s justification strategy capabilities and even, as a secondhand source, in epistemic thinking. Further within-person longitudinal study could reveal how students shift from one profile to another, or whether the profiles are mostly immutable. If further studies find student’s shifting from one profile to another during their studies, the structured learning diary could be used to measure this change at scale.

The present study employed a structured learning diary on the degree program level, over a set of tens of courses. Regular updates on any SLD results in extra workload for students. However, the main point is the return on the investment. The same SLD can visualize the whole degree programme, incorporate course links, facilitate diary activities, and note keeping. Moreover, the tool can double as a tool to inform teachers on how students experience learning the topics being taught, which, through adaptive teaching, can again benefit the learner. These large-scale benefits require implementation of such practices at scale, or selectively on capstone courses that aim to connect past learning to courses ahead.

Developers of learning technology tools should be informed by this study that students make meaningful updates on their structured learning diaries. The finding is limited to the intervention setting and sample described in this study. Perhaps the combination of a mind mapping type of visual approach and the freedom of writing natural language comments suited the context of engineering education particularly well. The intervention design permitted many levels of freedom for students to interact with the SLD while maintaining the possibility to extract and aggregate data from the files on a weekly basis. This approach, based on our experience, is worthwhile to consider in future instructional designs, intervention research designs, and by the developers of learning technology tools.

The results of this study manifest that university students use varying strategies when they justify their selections in an SLD tool. Moreover, the results show that there are meaningful epistemic cognitive processes behind the selections students make in structured learning diary. On the other hand, the results of this study suggest that a fair share of students do not seem to actively interact with the digital journaling tool presented in this study. These students seem to justify their diary content quite unilaterally, i.e., through personal justification or relying on authority sources.

6. Conclusion

Based on the results of this study, it can be concluded that university students use varying means of justification dimensions when justifying their structured learning diaries (SLD) content, as suggested by the previous literature. The three justification strategies identified were personal justification, justification by authority, and justification by multiple sources. All participants used personal justification, and three (PJ) used it exclusively when justifying their selection in the SLD tool. Students using justification by multiple sources in addition to personal justification (MS) associated with the most active use of structured items and revisiting and changing their difficulty estimates on course topics. Students employing all justification strategies (DS) in the interview were able and willing to reflect at an advanced level and reach conclusive resolutions on their internal dialog. However, they were not among the most active in using structured clickable items to reflect on their learning. This finding can contribute to the development of learning technology tools that at the same time advance learning and double as measure instruments (at scale). However, the findings introduced a new question for future studies: are the DS students less active in diary use because they make solid thoroughly processed decisions in their diaries, whereas the MS students are in a transforming stage on their way to more making more stable and justified (epistemic) decisions?

The results show promise in further explorations of the possibilities in using structured learning diaries as a tool to measure justification strategies, and more broadly, epistemic cognition. This conclusion is twofold. First, the diary content is meaningful for the students. Based on the interviews, most students justify their diary content on authority sources, such as course grades, or observations from multiple points of reference. Second, learning diary behavior is connected with students’ use of different justification strategies. Further studies are needed to study this relationship in more detail, in order to design more impactful structured learning diaries.

Data availability statement

The datasets presented in this article are not readily available because parts of the transcribed interviews may contain personal information. Requests to access the datasets should be directed to the corresponding author.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author contributions

VK and SL-Y contributed to the conception and design of the study. VK organized the data and facilitated the initial analysis. VK wrote the first draft of the manuscript. VK, SL-Y, and EK contributed the final analysis, inferences, and manuscript revisions. VK and EK prepared the final version of the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This study was partly funded by the Vietsch Foundation.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2023.1223732/full#supplementary-material

References

Barzilai, S., and Weinstock, M. (2015). Measuring epistemic thinking within and across topics: a scenario-based approach. Contemp. Educ. Psychol. 42, 141–158. doi: 10.1016/j.cedpsych.2015.06.006

CrossRef Full Text | Google Scholar

Bloom, B. S. (1953). Thought-processes in lectures and discussions. J. Gen. Educ. 7, 160–169. Available at: http://www.jstor.org/stable/27795429

Google Scholar

Bråten, I., Brandmo, C., and Kammerer, Y. A. K. (2019). A validation study of the internet-specific epistemic justification inventory with Norwegian Preservice teachers. J. Educ. Comput. Res. 57, 877–900. doi: 10.1177/0735633118769438

CrossRef Full Text | Google Scholar

Bråten, I., Ferguson, L. E., Stromso, H. I., and Anmarkrud, A. (2013). Justification beliefs and multiple-documents comprehension. Eur. J. Psychol. Educ. 28, 879–902. doi: 10.1007/s10212-012-0145-2

CrossRef Full Text | Google Scholar

Bråten, I., Ferguson, L. E., Strømsø, H. I., and Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: relations between epistemic cognition while Reading and sourcing and argumentation in essays. Br. J. Educ. Psychol. 84, 58–85. doi: 10.1111/bjep.12005

PubMed Abstract | CrossRef Full Text | Google Scholar

Bråten, I., Strømsø, H. I., and Samuelstuen, M. S. (2005). The relationship between internet-specific epistemological beliefs and learning within internet technologies. J. Educ. Comput. Res. 33, 141–171. doi: 10.2190/E763-X0LN-6NMF-CB86

CrossRef Full Text | Google Scholar

Bråten, I., Strømsø, H. I., and Samuelstuen, M. S. (2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemp. Educ. Psychol. 33, 814–840. doi: 10.1016/j.cedpsych.2008.02.001

CrossRef Full Text | Google Scholar

Broadbent, J., Panadero, E., and Fuller-Tyszkiewicz, M. (2020). Effects of Mobile-app learning diaries Vs online training on specific self-regulated learning components. Educ. Technol. Res. Dev. 68, 2351–2372. doi: 10.1007/s11423-020-09781-6

CrossRef Full Text | Google Scholar

Bromme, R., Kienhues, D., and Porsch, T. (2010a). “Who knows what and who can we believe? Epistemological beliefs are beliefs about knowledge (mostly) to be attained from others” in Personal Epistemology in the Classroom: Theory, Research, and Implications for Practice. eds. L. D. Bendixen and F. C. Feucht (Cambridge: Cambridge University Press), 163–194. doi: 10.1017/CBO9780511691904.006

CrossRef Full Text | Google Scholar

Bromme, R., Pieschl, S., and Stahl, E. (2010b). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological beliefs and metacognition. Metacogn. Learn. 5, 7–26. doi: 10.1007/s11409-009-9053-5

CrossRef Full Text | Google Scholar

Brownlee, J., Olafson, L., and Schraw, G. J. (2017). “Teachers’ personal epistemologies: evolving models for informing practice” in Current perspectives on cognition, learning and instruction. eds. D. H. Robinson and M. M. Griffin (Charlotte, NC: Information Age Publishing, Inc)

Google Scholar

Brownlee, J. L., Schraw, G., Walker, S., and Ryan, M. (2016). ““Changes in Preservice teachers’ personal epistemologies,” in Handbook of Epistemic Cognition. eds. J. A. Greene, W. A. Sandoval, and I. Bråten (Abingdon: Routledge).

CrossRef Full Text | Google Scholar

Buehl, M. M., Alexander, P. A., and Karen Murphy, P. (2002). Beliefs about schooled knowledge: domain specific or domain general? Contemp. Educ. Psychol. 27, 415–449. doi: 10.1006/ceps.2001.1103

CrossRef Full Text | Google Scholar

Cohen, L., Manion, L., and Morrison, K.. Research methods in education. 8th. London: Routledge, Taylor & Francis Group, (2018).

Google Scholar

Eastwood, J. L., Koppelman-White, E., Mi, M., Wasserman, J. A., Krug Iii, E. F., and Joyce, B. (2017). Epistemic cognition in medical education: a literature review. Int. J. Med. Educ. 8, 1–12. doi: 10.5116/ijme.5849.bfce

PubMed Abstract | CrossRef Full Text | Google Scholar

Elby, A. (2009). Defining personal epistemology: a response to Hofer & Pintrich (1997) and Sandoval (2005). J. Learn. Sci. 18, 138–149. doi: 10.1080/10508400802581684

CrossRef Full Text | Google Scholar

Fabriz, S., Ewijk, C. D.-V., Poarch, G., and Büttner, G. (2014). Fostering self-monitoring of university students by means of a standardized learning journal—a longitudinal study with process analyses. Eur. J. Psychol. Educ. 29, 239–255. doi: 10.1007/s10212-013-0196-z

CrossRef Full Text | Google Scholar

Ferguson, L. E., and Bråten, I. (2013). Student profiles of knowledge and epistemic beliefs: changes and relations to multiple-text comprehension. Learn. Instr. 25, 49–61. doi: 10.1016/j.learninstruc.2012.11.003

CrossRef Full Text | Google Scholar

Ferguson, L. E., Bråten, I., and Strømsø, H. I. (2012). Epistemic cognition when students read multiple documents containing conflicting scientific evidence: a think-aloud study. Learn. Instr. 22, 103–120. doi: 10.1016/j.learninstruc.2011.08.002

CrossRef Full Text | Google Scholar

Ferreira, P., Costa, A. M., Simão, V., and Lopes da Silva, A. (2014). Does training in how to regulate One’s learning affect how students report self-regulated learning in diary tasks? Metacogn. Learn. 10, 199–230. doi: 10.1007/s11409-014-9121-3

CrossRef Full Text | Google Scholar

Geiser, C., Götz, T., Preckel, F., and Freund, P. A. (2017). States and traits: theories, models, and assessment. Eur. Assoc. Psychol. Assess. 33, 219–223. doi: 10.1027/1015-5759/a000413

CrossRef Full Text | Google Scholar

Greene, J. A., Azevedo, R., and Torney-Purta, J. (2008). Modeling epistemic and ontological cognition: philosophical perspectives and methodological directions. Educ. Psychol. 43, 142–160. doi: 10.1080/00461520802178458

CrossRef Full Text | Google Scholar

Greene, J. A., Cartiff, B. M., and Duke, R. F. (2018). A Meta-analytic review of the relationship between epistemic cognition and academic achievement. J. Educ. Psychol. 110, 1084–1111. doi: 10.1037/edu0000263

CrossRef Full Text | Google Scholar

Greene, J. A., Sandoval, W. A., and Bråten, I.. "Handbook of epistemic cognition." (2016).

Google Scholar

Greene, J. A., and Seung, B. Y. (2014). Modeling and measuring epistemic cognition: a qualitative re-investigation. Contemp. Educ. Psychol. 39, 12–28. doi: 10.1016/j.cedpsych.2013.10.002

CrossRef Full Text | Google Scholar

Hofer, B. K. (2006). Beliefs about knowledge and knowing: integrating domain specificity and domain generality: a response to Muis, Bendixen, and Haerle (2006). Educ. Psychol. Rev. 18, 67–76. doi: 10.1007/s10648-006-9000-9

CrossRef Full Text | Google Scholar

Hofer, B. K. (2016). “Epistemic cognition as a psychological construct” in Chap. Epistemic cognition as a psychological construct in handbook of epistemic cognition. eds. J. A. Greene, W. A. Sandoval, and I. Bråten (New York: Routledge), 31–50.

Google Scholar

Hofer, B. K., and Pintrich, P. R. (1997). The development of epistemological theories: beliefs about knowledge and knowing and their relation to learning. Rev. Educ. Res. 67, 88–140. doi: 10.3102/00346543067001088

CrossRef Full Text | Google Scholar

Hofer, B. K., and Sinatra, G. M. (2010). Epistemology, metacognition, and self-regulation: musings on an emerging field. Metacogn. Learn. 5, 113–120. doi: 10.1007/s11409-009-9051-7

CrossRef Full Text | Google Scholar

Johanes, P. (2017). “Epistemic cognition: a promising and necessary construct for enriching large-scale online learning analysis” in Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale. eds. C. Urrea, J. Reich, and C. Thille (Cambridge, Massachusetts, USA: Fourth Annual ACM Conference on Learning at Scale). doi: 10.1145/3051457

CrossRef Full Text | Google Scholar

Johnson, R. B., Onwuegbuzie, A. J., and Turner, L. A. (2007). Toward a definition of mixed methods research. J. Mixed Methods Res. 1, 112–133. doi: 10.1177/1558689806298224

CrossRef Full Text | Google Scholar

Kammerer, Y., Amann, D. G., and Gerjets, P. (2015). When adults without university education search the internet for health information: the roles of internet-specific epistemic beliefs and a source evaluation intervention. Comput. Hum. Behav. 48, 297–309. doi: 10.1016/j.chb.2015.01.045

CrossRef Full Text | Google Scholar

Kammerer, Y., Gottschling, S., and Bråten, I. (2021). The role of internet-specific justification beliefs in source evaluation and corroboration during web search on an unsettled socio-scientific issue. J. Educ. Comput. Res. 59, 342–378. doi: 10.1177/0735633120952731

CrossRef Full Text | Google Scholar

Kitchener, K. S. (1983). Cognition, metacognition, and epistemic cognition. Hum. Dev. 26, 222–232. doi: 10.1159/000272885

CrossRef Full Text | Google Scholar

Kivimäki, V., Pesonen, J., Romanoff, J., Remes, H., and Ihantola, P. (2019). Curricular concept maps as structured learning diaries: collecting data on self-regulated learning and conceptual thinking for learning analytics applications. J. Learn. Anal. 6, 106–121. doi: 10.18608/jla.2019.63.13

CrossRef Full Text | Google Scholar

Kivimaki, V., Romanoff, J., and Remes, H. (2021). “Improvement of teaching and learning experience by online course diary tool: Qualitative exploration of teachers’ perspective” in Proceedings of the European University Information Systems Conference 2021 (EUNIS21). eds. S. Bolis, J-F. Desnos, L. Merakos and R. Vogl (EPiC Series in Computing Vol. 78, pp. 115–125). EasyChair Publications. doi: 10.29007/6z3k

CrossRef Full Text | Google Scholar

Klug, J., Ogrin, S., Keller, S., Ihringer, A., and Schmitz, B. (2011). A Plea for self-regulated learning as a process: Modelling, measuring and intervening. Psychol. Sci. 53:51.

Google Scholar

Lonka, K., Ketonen, E., and Vermunt, J. D. (2021). University students’ epistemic profiles, conceptions of learning, and academic performance. High. Educ. 81, 775–793. doi: 10.1007/s10734-020-00575-6

CrossRef Full Text | Google Scholar

Lunn Brownlee, J., Ferguson, L. E., and Ryan, M. (2017). Changing Teachers' epistemic cognition: a new conceptual framework for epistemic reflexivity. Educ. Psychol. 52, 242–252. doi: 10.1080/00461520.2017.1333430

CrossRef Full Text | Google Scholar

Lunn Brownlee, J., Rowan, L., Ryan, M., Walker, S., Bourke, T., and Churchward, P. (2019). Researching teacher Educators' preparedness to teach to and about diversity: investigating epistemic reflexivity as a new conceptual framework. Asia Pac. J. Teach. Educ. 47, 230–250. doi: 10.1080/1359866X.2018.1555794

CrossRef Full Text | Google Scholar

Mackey, A., and Gass, S. M.. Second language research: methodology and design. (2011). Mahwah, NJ: Lawrence Erlbaum, 2005.

Google Scholar

Marcus, K. (2021). Meta-reflexivity and epistemic cognition in social science teacher education. J. Soc. Sci. Educ. 20:4087. doi: 10.11576/jsse-4087

CrossRef Full Text | Google Scholar

Mason, L., Ariasi, N., and Boldrin, A. (2011). Epistemic beliefs in action: spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learn. Instr. 21, 137–151. doi: 10.1016/j.learninstruc.2010.01.001

CrossRef Full Text | Google Scholar

Mason, L., Boldrin, A., and Ariasi, N. (2010). Searching the web to learn about a controversial topic: are students Epistemically active? Instr. Sci. 38, 607–633. doi: 10.1007/s11251-008-9089-y

CrossRef Full Text | Google Scholar

Mason, L., and Bromme, R. (2010). Situating and relating epistemological beliefs into metacognition: studies on beliefs about knowledge and knowing. Metacogn. Learn. 5, 1–6. doi: 10.1007/s11409-009-9050-8

CrossRef Full Text | Google Scholar

McComas, W. F., and Burgin, S. R. (2020). A critique of “STEM” education. Sci. & Educ. 29, 805–829. doi: 10.1007/s11191-020-00138-2

CrossRef Full Text | Google Scholar

McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochem. Med. 22, 276–282. doi: 10.11613/BM.2012.031

PubMed Abstract | CrossRef Full Text | Google Scholar

Muis, K. R. (2008). Epistemic profiles and self-regulated learning: examining relations in the context of mathematics problem solving. Contemp. Educ. Psychol. 33, 177–208. doi: 10.1016/j.cedpsych.2006.10.012

CrossRef Full Text | Google Scholar

Muis, K. R., Bendixen, L. D., and Haerle, F. C. (2006). Domain-generality and domain-specificity in personal epistemology research: philosophical and empirical reflections in the development of a theoretical framework. Educ. Psychol. Rev. 18, 3–54. doi: 10.1007/s10648-006-9003-6

CrossRef Full Text | Google Scholar

Onwuegbuzie, A., and Collins, K. (2007). A typology of mixed methods sampling designs in social science research. Qual. Rep. 12, 281–316. doi: 10.46743/2160-3715/2007.1638

CrossRef Full Text | Google Scholar

Panadero, E., Klug, J., and Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scand. J. Educ. Res. 60, 723–735. doi: 10.1080/00313831.2015.1066436

CrossRef Full Text | Google Scholar

Perry, G. W. (1970). Forms of intellectual and ethical development in the college years: a scheme. Jossey-bass higher and adult education series. New York: Holt, Rinehart and Winston.

Google Scholar

Pesonen, J. A., Ketonen, E. E., Kivimäki, V., and Ihantola, P. (2020). Does using structured learning diaries affect self-regulation or study engagement? An experimental study in engineering education IEEE.

Google Scholar

Royce, J. R. (1967). “Metaphoric knowledge and humanistic psychology” in Challenges of humanistic psychology. ed. J. T. F. Bugental (New York, NY: McGraw-Hill), 21–28.

Google Scholar

Sammons, P.. "Sage handbook of mixed methods in Social & Behavioral Research." Thousand Oaks, California: SAGE Publications, Inc., (2010).

Google Scholar

Sandoval, W. A., Greene, J. A., and Bråten, I. (2016). Understanding and promoting thinking about knowledge: origins, issues, and future directions of research on epistemic cognition. Rev. Res. Educ. 40, 457–496. doi: 10.3102/0091732X16669319

CrossRef Full Text | Google Scholar

Schmitz, B., and Perels, F. (2011). Self-monitoring of self-regulation during math homework behaviour using standardized diaries. Metacogn. Learn. 6, 255–273. doi: 10.1007/s11409-011-9076-6

CrossRef Full Text | Google Scholar

Schmitz, B., and Wiese, B. S. (2006). New perspectives for the evaluation of training sessions in self-regulated learning: time-series analyses of diary data. Contemp. Educ. Psychol. 31, 64–96. doi: 10.1016/j.cedpsych.2005.02.002

CrossRef Full Text | Google Scholar

Schraw, G., Olafson, L., and Lunn Brownlee, J., and M. Vander Veldt Brye. "Reflection and reflexivity: a focus on higher order thinking in Teachers' personal epistemologies." G. Schraw, L. Olafson, J. Lunn Brownlee, and M. Vander Veldt Brye, (eds) Teachers' personal epistemologies: evolving models for informing practice (Current Perspectives on Cognition, Learning and Instruction) 409–427. Charlotte, NC: United States of America: Information Age Publishing, (2017).

Google Scholar

Sime, D. (2006). What do learners make of Teachers' gestures in the language classroom? Int. Rev. Appl. Linguist. Lang. Teach. 44, 211–230. doi: 10.1515/IRAL.2006.009

CrossRef Full Text | Google Scholar

Tashakkori, A., and Creswell, J. W. (2007). Editorial: the new era of mixed methods. J. Mixed Methods Res. 1, 3–7. doi: 10.1177/2345678906293042

CrossRef Full Text | Google Scholar

Tashakkori, A., and Teddlie, C. (2016). Handbook of Mixed Methods in Social & Behavioral Research. 2nd ed. (Thousand Oaks, California: SAGE)

Google Scholar

Teddlie, C., and Sammons, P. (2010). Applications of mixed methods to the field of educational effectiveness research. London: Routledge, 129–166.

Google Scholar

Urbaniak, G. C., and Plous, S. (2013). Research randomizer (version 4.0) [computer software]. Available at: http://www.randomizer.org/ (Accessed June 8, 2023).

Google Scholar