The determinant factors of medical students' learning behavior in the national medical competency examination in Indonesia: A qualitative study

Indonesia has applied for a national competency exit examination for medical graduates since 2014. This examination has an impact on students. There is limited literature that describes the impact mechanism of the national examination. This study aimed to identify the determinant factors of the learning behavior of medical students in the national medical competency examination during the COVID-19 pandemic in Indonesia. This qualitative study was conducted using semi-structured in-depth face-to-face interviews with 13 medical students from the Faculty of Medicine Universitas Islam Malang and online interviews with 13 medical students from the Faculty of Medicine Universitas Gadjah Mada, Indonesia. Theoretical sampling was carried out by including three medical students from a faculty of medicine in east Indonesia. Semi-structured in-depth interviews with respondents from FK Unisma were conducted offline with strict health protocols. Meanwhile, communications with respondents from FK-KMK UGM and the eastern Indonesia medical faculty were conducted online by telephone or through Zoom meetings. Transcribed interviews were coded openly. Themes were finalized through discussion to reach a consensus. The extracted concepts from our study were classified into 10 categories that describe the determinant factors of learning behavior and two categories of learning behavior. These categories include task value, goal orientation, weakness, impact failure, benefits of passing the examination, self-efficacy, beliefs, barriers to learning, environment factors, and test anxiety. The learning behavior itself consists of two categories: quality of learning and regulation of learning. The conclusion is determinant factors consisting of internal and external factors. Internal factors include task value, goal orientation, weakness, benefits of passing the examination, self-efficacy, belief, and test anxiety. The external factor includes the environmental factor. These factors determine the medical student learning behavior in the national medical competency examination during the COVID-19 pandemic in Indonesia, both in quality and regulation of learning. The impact of failure only determines the regulation of learning, while the barrier to learn only determines the quality of learning.

Introduction

National medical competency examinations have been conducted in many countries in the world to ascertain that the competencies of graduating medical doctors can fulfill the minimal required standard. Indonesia has been conducting the national medical competency examinations known as the Indonesia Medical Doctor National Competency Examination (IMDNCE) since 2014 based on Indonesian Medical Act No. 20/2013, which consists of two components. The IMDNCE in Indonesia language is called as UKMPPD. There are multiple-choice questions using computer-based testing methods (MCQs-CBT) to assess candidates' knowledge, and Objective Structured Clinical Examinations (OSCE) to assess candidates' clinical skill performance. The IMDNCE is high-stakes assessment that has been widely utilized as an evaluation tool in medical education, apart from the main function of assuring patient safety (Melnick, 2009; Dillon et al., 2013; Archer et al., 2016). These functions of the assessment always need to be evaluated in the implementation process to prevent malpractice in the assessment (Shumway and Harden, 2003; Baartman et al., 2006; Norcini et al., 2011, 2018).

There are many aspects that must be considered in the implementation of assessment, including validity, reliability, impact on the learner and educational program, and practicability, as well as assessments that have a catalytic effect (Shumway and Harden, 2003; Baartman et al., 2006; Norcini et al., 2011, 2018). These assessment studies often only focus on psychometric studies involving validity and reliability testing related to the quality of the assessment (Schuwirth and Van der Vleuten, 2004a). Meanwhile, there are other important aspects closely connected to the impact of assessments on the learning and education process (Shumway and Harden, 2003; Van der Vleuten and Schuwirth, 2005; Norcini et al., 2011, 2018; Schuwirth and Van der Vleuten, 2011).

The intended impact of the assessment on student learning is related to the educational impact of assessment (Schuwirth and Van der Vleuten, 2004b), also known as “the testing effect”, “consequential validity”, “test enhanced learning”, “backwash”, “washback”, and “testing phenomenon” (Cilliers et al., 2010), which include important elements in the utility of the assessment system (Van der Vleuten and Schuwirth, 2005). Assessment can also encourage students to be able to carry out a variety of learning activities with various kinds of metacognitive regulations (Knight, 2001; Al Kadri et al., 2009, 2011; Firmansyah, 2015). This confirms that assessments have a very strong relationship to learning behavior so that the term “assessment-driven learning” appears currently “in vogue” (Norcini et al., 2011). The magnitude of the assessment impact on changes in student learning behavior is very strongly related to the magnitude of the consequences of the assessment that will occur (Al Kadri et al., 2011; Cilliers et al., 2012a,b,c). The student learning behavior is influenced by an appraisal of the impact of assessments and appraisal of their learning responses (Cilliers et al., 2010, 2012a,b,c; Cilliers, 2015). In theory, several factors as external factors that can influence learning behavior have not been clearly studied, such as curriculum design, student workloads, and learning methods (Broekkamp and Van Hout-Wolters, 2007). This is a reality that assessment does not the only factor in encouraging students to learn. However, there are other determinant factors that function as a mechanism factor for learning behavior change in students.

Meanwhile, the known determinant factors of learning are those related to students' perceptions of the assessment, environmental factors, and contextual factors. These contextual factors consist of sociocultural, work opportunities, and the healthcare system. A very complex relationship occurs between these factors (Leung et al., 2008; Al Kadri et al., 2009, 2011; Al-Kadri et al., 2012; Moghaddam et al., 2019). A previous study focuses on accommodating elements of culture and the learning environment in an assessment program but is limited to the context of summative assessment, not specific in high-stakes assessment, and carried out in countries outside Indonesia that have very different sociocultural conditions, due to differences in ethnicity, religion, customs, and geographical location.

In addition, the factors of differences in cultural elements and the learning environment still need to be explored more deeply concerning their influence on student learning behavior (Al-Kadri et al., 2012). Extensive research related to assessment-driven learning has been conducted; however, there are limited studies that comprehensively and completely review how summative assessments impart a measurable impact on student learning, especially in high-stakes contexts and different cultural contexts (Norman et al., 2010). High-stakes assessment has only been a recently explored part of the curriculum design, not specifically in the learning impact. Accordingly, to confirm the findings from previous studies, this study was conducted to explore the determinant factors of learning behavior as a possible model for the mechanism of learning preparation before the MCQs-CBT IMDNCE as a cognitive high-stakes assessment for medical students in Indonesia. It is expected the mechanism of assessment-driven learning could be understood clearly in different contexts and cultures. By knowing the mechanism impact of assessment that drives learning, it is hoped that we will strengthen the mechanism impact that has positive drivers and minimize negative factors. This study was also to validate the internal and external factors that mediate student approaches on learning in a different context of assessment (high-stakes assessment) and different cultures (Indonesia context in the pandemic era).

Methods

This study employed a qualitative methodology using the constructivist grounded theory (GT) approach (Corbin and Strauss, 2008). This is a suitable approach to discover new areas of the subject under study or to explore a known area from a new perspective. Another reason for using the GT approach is the discovery of process and social interactions. Data collection was carried out by conducting in-depth interviews with 29 respondents of medical students who had completed clinical rotation and would have taken the cognitive national licensure examination.

This study was conducted at the Faculty of Medicine Universitas Islam Malang (FK Unisma), Faculty of Medicine, Public Health and Nursing Universitas Gadjah Mada (FKKMK UGM), and the Faculty of Medicine in eastern Indonesia. The main researcher (MIF) has a relationship with participants from FK Unisma as a medical educator, but not in other institutions. Respondents were selected by taking maximum variation sampling at two medical education institutions, one with an A accreditation level that implemented competency-based curriculum earlier (public medical school FK-KMK UGM) (n = 13), and the other with a B accreditation level that implemented competency-based curriculum recently (private medical school, Faculty of Medicine Universitas Islam Malang) (n = 13). Both the institutions were from the west region of Indonesia. The criteria of respondents were male and female students with low-, medium-, and high-grade performance averages (GPAs). Meanwhile, to obtain a theoretical sampling, three respondents were added from another Indonesian region (the eastern Indonesia Medical Faculty) (n = 3) to ensure the representativeness of the east region Indonesia with different sociocultural conditions. So, the total number of respondents was 29.

Ethical approval was issued by the Medical and Health Research Ethics Committee (MHREC) of the FKKMK UGM, Dr. Sardjito General Hospital, Indonesia, with file number KE/FK/0779/EC/2020. All participants signed the informed consent forms regarding their agreement to participate according to the guidelines of brief descriptions to study subjects from the MHREC of the FKKMK UGM, in Yogyakarta, Indonesia.

Because of mobility restriction due to the COVID-19 pandemic, data collection was conducted through offline and online interviews depending on the location of the respondent's residence from August 2020 to January 2021. Semi-structured in-depth interviews with the respondents from FK Unisma were conducted offline with strict health protocols. Meanwhile, communications with respondents from FK-KMK UGM and the eastern Indonesia medical faculty were conducted online by telephone or Zoom meetings. We conducted semi-structured in-depth interviews to stimulate the participants to talk freely about their experiences when preparing for the national licensing examination. All participants received an explanation about research goals, reason for doing the research, and the semi-structured in-depth interview process and gave their consent to participate. The semi-structured in-depth interviews were conducted using guiding questions, as shown in Appendix 1. The list of questions was based on the previous research with some adaptations (Firmansyah, 2015). The list of questions was validated by two experts with medical education backgrounds. All interviews were audio-recorded using a digital recorder after all participants signed an agreement with verbal informed consent. The main researcher (MIF), who was also the interviewer, was a lecturer in FK Unisma who graduated with a Master degree in Medical Education had training in qualitative research training and previously had also conducted qualitative research. The coauthor (YS and GRR) is an expert in medical education assessment with experience in quantitative and qualitative research in medical education.

The semi-structured in-depth interviews were conducted one by one to groups of participants. We conducted in depth interviews with the participants from FK Unisma at first to obtain initial data for the study material for the next group interview. Then proceeded with in-depth interviews with group participants from FK-KMK UGM who have not been previously known to check whether there are differences in the data from the previous group participants and to obtain variations in the data that may occur. We aimed to find out the consistency of the themes or categories from the data of the previous group of participants. Data saturation occurred after the 21st participant. After that, we re-checked the data saturation that had occurred by including three participants from the eastern Indonesia Medical Faculty with the consideration that there were differences in demographic and cultural characteristics.

The first author (MIF) conducted semi-structured in-depth interviews with offline and online interviews and recorded them by an audio recorder. The recording was performed using a special recording device (Digital Voice Recorder VN-7111PC). At the time of the interview, we also made brief notes related to typical or important information in the form of field notes for brief evaluation materials in preparation for the next interview. Offline interviews were conducted in the classroom; meanwhile, online interviews were conducted from home via telephone communication. All interviews took 40–80 min for each participant and were transcribed verbatim by a transcriber service agent. MIF examined all transcripts by checking line by line to see the congruence between the transcript and recordings. This process was supervised by two other authors (GRR and YS), who are healthcare professionals and have experience in conducting qualitative research and have experience and expertise in student assessments. All audio records were listened to several times. In addition, we also observed the respondents' tone of voice and face directly, but the non-verbal responses from the face were not so visible because the participants were wearing a mask. For some participants who were willing to use Zoom, the non-verbal responses could be observed. To increase trustworthiness, member checking was also carried out. MIF analyzed the transcripts repeatedly and rigorously under the supervision of GRR and YS. Data saturation was achieved when there were no new comments from the participants and agreed on by MIF, GRR, and YS.

Data analysis

Data analysis was conducted on the transcripts of 29 respondents. Data coding was performed by three different coders. Transcripts were read and open coded by three coders: MIF with the help of independent coders (MER and TAR). MER was the independent coder for transcripts nos. 1–14, and TAR was the independent coder for the remaining transcripts (nos. 15–29). MER and TAR are healthcare professionals with experience in conducting qualitative data analysis. Transcription was independently coded line by line by these three coders. Data codification was performed using three different identifiers with the use of thematic content analysis, which produces subcategories as a coding result. After getting the coding results, the researchers performed axial coding, namely, grouping the predetermined codes into categories. Differences in codification were debated and resolved through various discussions by the three coders. Through the discussion and debate, the identified subcategories, categories, and their associations that were clustered into themes were explored. In the process, the representative quotations were also discussed and selected. Data display was carried out by grouping based on thematic content analysis, which was shown as a fishbone model in Figure 2. Data display was also verified with independent coders and two experts in medical education (GRR and YS).

Data display described the coding result as subcategories and categories as determinant factors of learning behavior. All the identified subcategories, categories, and themes are presented in Table 1. Data verification and credibility were achieved by conducting member checking and triangulation. Member checking was conducted by providing participants with the transcripts to make sure that the result of in-depth interview is suitable as intended by participants. Trustworthiness also was conducted by GRR and YS, which helped clarify the results of codification made by the three coders. Differences in the codification were then debated by GRR and YS and resolved through discussion until a consensus was reached. The subcategories, main categories, and representative quotations were also reviewed and revised by GRR and YS. No new categories and subcategories emerged after 21 semi-structured in-depth interview transcripts were analyzed. The remaining transcripts were checked and used to ensure data saturation. Memos and documents from the interviews and coding steps, which were kept by the first author, were also used to increase credibility. Triangulation of the data in this study was made by inviting the respondents from three groups of medical students, that is, public medical school, private medical school, and east Indonesia medical school, with variations in GPA and culture. GRR and YS also provided expert checking to determine whether the coding and representative quotations completely described the category and subcategory that were represented.

TABLE 1

Table 1. Classification of theme, categories, and subcategories.

Results

We selected respondents with certain criteria, as shown in Table 2, to obtain diverse data. The criteria were the type of medical education institution, that is, public institutions with an A accreditation level and private institutions with a B accreditation level, gender, and additional medical education institutions located in eastern Indonesia. Each group consisted of men and women with low, medium, and high GPA variations. The characteristics of the participants are shown in Table 2.

TABLE 2

Table 2. Distribution of participants based on faculty of medicine, gender, and academic performance (GPA).

The extracted concepts from our data are presented as a model in Figure 1. The impact of high-stakes assessment on learning was mediated through various determinants of action. We classified our findings into 10 categories determinant factors of learning behavior. The determinant factors are internal factors that included task value, goal orientation, weakness, impact failure, benefits of passing the examination, self-efficacy, beliefs, barrier to learn, and anxiety, and external factors that included environment factors consisting of institution regulation and peer influence, as shown in Figure 2. The learning behavior itself consists of two categories: quality of learning and regulation of learning. The linkage between the determinant factors of learning and learning impact is shown in Table 3.

FIGURE 1

Figure 1. Suggested model for learning impact of high stake assessment in the individual, interpersonal, and community levels.

FIGURE 2

Figure 2. Classification of theme, categories, and subcategories as determinant factors of learning behavior.

TABLE 3

Table 3. Linking determinant factor of learning and learning impact.

Task value

The task value is a way of looking at the examination, which shows the examination is important or not, which will reflect in a students' attitude and actions toward their efforts to complete the examination well. This task value can be observed in the response of W5:

[Quote 1]“...Answer: No Doc, at least that's the one, Doc, reading comics, reading novels, or playing games. But usually if I do that, there will be a bad feeling.

Question: Because?

Answer: Why did I even–, yes, because of me, I should be more focused, focus on what is called UKMPPD learning, because UKMPPD is important, but instead I spend time playing games or reading novels, or read comics....” W5

The students have the perception that the national examination is something important so that it can encourage students and even force students to study. This was stated by W1 and R12:

[Quote 2] “...of course, because UKMPPD is the only one that determines whether or not a doctor is eligible to graduate, so it's like after real school for 6 years was determined at that time only with the UKMPPD....” W1

[Quote 3] “...But as soon as I take UKMPPD, like it or not, I have to force myself to study, to read where I am lacking, like that...” R12

Goal orientation

Goal orientation is an orientation that is an individual's reason for trying, which includes processes and behaviors, to achieve or obtain certain goals. This can be seen in responses of R13 and W11, which consist of performance and learning goals:

[Quote 4] “Answer:.......if for me it's not just passing, but also the score is above 80. Actually, if this UKMPPD passes, it's also worth considering if you go to school again. I want to graduate with satisfactory grades.

Question: What did you do to achieve that?

Answer: to achieve that what I do is when getting closer to UKMMPD, I initially thought that I could study on my own, but it turns out that I can't be. So we have to study together…” R13

[Quote 5] “..... If I were to be able to pass it I was sure I would pass. So, actually my motivation is to get knowledge, so how can I get a lot of knowledge. If it's to pass, isn't it–, okay, from the start, God willing, I'm sure that just to pass, the pass rate is already, I'm sure i'll pass, that's what it is. So yesterday, my motivation was how I could really understand a lot of knowledge from this UKMMPD preparation, like that, Doc......” W11

Weakness

Some students in preparing for examinations feel weak in memorizing the learning material, which can cause them to fail the examination. With this vulnerability, students will carry out the memorization learning process, as stated by R12:

[Quote 6] “.... Whether there is a tutor from the campus is also quite helpful. So more often to the reviews again. Keep learning a lot, the same as yesterday when I tried to write it down in a small book, it's like making sticky notes like that, Doc. So pasted on the wall of the room. When you wake up or want to sleep, read it by rote. So it's like memorizing therapy, like that, Doc. Because it's a lot. I feel I lack in that topic so I learn more about that topic....” (R12)

Students also create study groups because they felt that there was material that had been missed for a long time, as stated by W11:

[Quote 7] “……Well but yesterday we made this, Doc, –what?–, small study group. So at that time, we started from the chapter, –what?–, because clinical rotation in internal medicine and pediatrics has been done for a long time, so we started to study that topics, we made a small group of about 5 people, we started to pay attention to discuss the answer exercises.....” W11

Impact of failure

Students will prepare for their examinations by studying hard because there is some fear if they do not pass the examination—their study will be lengthened, so they will also pay more tuition fees and they will lose their classmates, which can cause a decrease in their enthusiasm to learn. That is why students avoid it as stated by R8 and W8:

[Quote 8] “....Because if you want to say I'm afraid, I've been preparing for a long time. What I'm afraid of if I don't pass. I think if I didn't graduate, it meant I didn't go home, lived in Malang, that's what it was. Alone in the boarding house, I see.” R8

[Quote 9] “.....Yes. Overload–, this burden anyway, Doc. Because I said earlier, ee–, after the clinical study, it tends to be group study, that's what it is. And I feel that if, for example, ee–, for example, if you don't graduate alone, you won't have any friends to study with, which in the end, uh–, becomes less motivated, less enthusiastic, Doctor....” W8

Benefits of passing the examination

Some students have a perception that the benefits of the examination are as a means of preparing for the internship program or preparing to work when becoming a doctor. Students can also continue to further professional training, such as specialist, Master's program, as stated by R4:

[Quote 10]”... “First, I have a big picture, the big picture. Oh, the medical education that I have to achieve, it turns out that I have to master it like this. Very different when I first entered the pre-clinic. What I received was material, then maybe lab skills like that, doctor. But I don't know which way I should go. When it started to clerkship oh already started to be arranged a little. When I prepare the national exam, oh, I knew competency that I have to master it, for example, third line management. So, when I was at the Puskesmas I had this disease, I had to treat it like this.”.......and from the information i got was that if we want to continue our studies, there are some scores from this UKMPPD that are used as a reference when we want to continue our education. For example, if I want to take a surgical specialist, my test scores automatically when the UKMPPD CBT or OSCE should have higher points in surgical cases than non-surgical cases. Or the points are better than the others,.....” R4

Self-efficacy

Self-efficacy is a student's self-confidence that the student will be able to fulfill a task/obligation well. Students with high self-efficacy will design their learning process so that they can be successful and can pass the examination. The more learning processes that have been carried out, the better the self-efficacy. According to the results of in-depth interviews in this research, there is a reciprocal relationship between self-efficacy and learning behavior as stated by W1:

[Quote 11] “......If it's from friends who are studying, it's for sure, if friends are not studying, doc, if we have to do one shoot, isn't it, because yes, no one will know what the problem is, but we ourselves already feel it's okay. Studying maximally, you can definitely do that, it's like motivating yourself if you can definitely do that....” W1

Student learning by following tutoring outside the campus is mostly due to lack of confidence, as stated by W8:

[Quote 12] “……But many of them may fail or are less optimal due to lack of self-confidence, maybe other friends may even take the tutoring, don't you think–. I also asked, Doc, yes, 'Why do you take so many tutoring?', that's right. 'There's already one tutoring that is intensive', so. 'Yes, let's just be more confident', that is. So I concluded, –what is it?–, it turns out that it was because of the lack of self confidence, Doc.......” W8

There are also students who have fairly high self-confidence, which is known from their decision not to participate in tutoring, as experienced by W8:

[Quote 13] “...challenge yourself so you don't get too, eh–, what is it?–, go with the flow, because, uh–, yesterday there were many who actually took lessons up to the 5th tutoring, and that not just one person, Doctor, there are many. And finally, I convinced myself that I didn't have to go along with it, what was that? It's just an impulsive person, I see. Because, –what is it?–, if we join the impulsiveness in the end, eh–, it's not really that effective. What's really effective is convincing yourself, then the second one is really preparation, ee–, that's really ready.” W8

Beliefs

Spiritual belief

Students choosing learning resources was also influenced by their spiritual beliefs because currently, there are many learning resources, especially practice questions from various kinds of tutoring circulating in cyberspace. To use learning resources in the form of questions, students believe they must get permission from the owner of the questions. This was followed so that learning activities were approved by God Almighty, as stated by W8:

[Quote 14] “..... we think about the blessing like that. I mean, is this allowed, actually, it's allowed to be used other than what was taught there, so yesterday we asked for permission as well, is it okay or not. It turns out that there are lessons that don't allow for example–, was it yesterday? “Ma^****” it doesn't allow for. So, we ourselves are not pleased, Doc, that's how it is. Then if it's “In^*****” who allows it, then what else, yesterday, I kind of forgot. So, we often use In^*****,…” W8

Students' spiritual beliefs accompany all their activities in preparation before the examination, both in learning and in non-learning activities, as followed by respondent R11. They believe that besides studying hard, they also accompany it by praying to God.

[Quote 15]”..... “...Ee–, because we also believe that even effort without prayer will not produce anything, from ourselves. So we have to try our best, study but also pray and we also believe, ee–, the power which is called, -what is it?-, invisible. There will always be help when we pray for others, if we are happy, we want others to be successful with us, there will be additional, additional–, what is the term?-, sustenance that does not know where so that which makes us sure that besides this effort also has an effect....” R11

Normative belief

Some students also have a normative belief that if you try hard, you will succeed, and if you master the patterns of the task type, you will succeed, as stated by T2:

[Quote 16] “......Well, and regarding, the task on UKMPPD questions, in general, is just something like that. The patterns are just like that, just like that. So, the more we master the theory and we master the patterns of task type, we can definitely answer the exam.” T2

Barriers to learn

Students' perceptions of themselves and their environment that hinder them from learning can become a mechanism for changing the students' learning behavior. For example, learning conditions during the COVID-19 pandemic by online learning often make them feel dizzy because they are in front of the computer for a long time, and lack of interaction with their lecturers or friends to face-to-face discussion, as stated by R7. However, some students prefer to study online during this pandemic, as stated by R13. He felt that with online learning, he did not have to go to campus for in-class learning.

[Quote 17] “...Because of we have to look at laptops– laptops for –what?– can be up to 5-6 hours a day or even more, yes, Doctor. So those eyes–what?– get tired easily, Doctor. Now, while I sometimes get migraines when my eyes get tired, so it's already– the quality for studying has decreased, Doctor. So” R7

[Quote 18] “…..If I were a doctor, the zoom meeting doesn't take much energy. If it's offline, we need to go to campus or to the guidance center. Online learning with Zoom is more profitable. So, the pandemic has no effect......” R13

Environmental factors

Environmental factors that drive students to carry out the learning process was peer influence and institutional influence.

Peer influence

Peer influence was also a driver of students learning in the community. Having a friend's advice in their environment who advised them to study hard to pass the examination was also a reflection of the peer influence, as stated by R3:

[Quote 19] “...Oh, yes, doctor. Yes, maybe because to be honest, when we were in the boarding house, doctor, when our friends who had graduated saw that we might be lazy or something, they often mocked us but in a joking way, that's what it is. So it's like, “Come on, you will take the exam, the exams is not finished. You didn't pass, blah, blah, blah”. So they often, more often, frighten and speak ill of us, doctors. That might motivate us in the same group house to be better than that one....” R3

Student learning by following tutoring outside the campus was mostly influenced by their friends, as stated by W11 and R8:

[Quote 20] “……All my close friends join outcampus tutoring, so that's it, so I join too, Doc. And, my friends said, 'Well, that's okay we join outcampus tutoring, because they say that we will get a lot of knowledge'. It means that we cann't get as much as possible if we don't join outcampus tutoring, we can do that...” W11

[Quote 21] “.......Usually, especially for us, we look at our friends. Our friend, oh, that friend follows the outcampus tutoring, for example, following this condition, we also want to join in outcampus tutoring, so, we don't feel–, feel–, feel like nervous or feel lost, if I don't learn, it is ok to be calm,....” R8

Institutional regulation

Institutional influence could drive student learning. Institutional conditions that require students to take tutoring on campus can make students come and study according to the schedule. This was conveyed by R8:

[Quote 22] “...But if the person who takes the tutoring, I think he has to be sure to learn it, even if he has to, because the schedule is really tight. In the end, in the first time we originally not intended to study, we have to study, that's it.” R8

Test anxiety

Students feel anxious after they have perceptions about the impact of failure in the assessment. Feelings of worry/excitement/anxiety before the examination can be a factor in the mechanism for changing learning behavior. With this feeling, students will prepare by learning more optimally, as conveyed by R7. In addition, students prefer to take a rest for a while to calm down and start learning again after that, as conveyed by R6 and R13:

[Quote 23] “….Very scary for me. For me, it's still– because I'm an anxious person, yes, Doctor, yes, so I think it's –what?–is that scary, Doctor. Because it has to be prepared in advance, yes, even in a matter of months, I see. Because indeed I also have to study extra, not just once or twice, even four times, five times I can only catch it,” R7

{Quote 24] “....When I'm nervous, doctor, I actually prefer to calm him down, in the sense that I'm quiet first. I didn't do anything first for a full day, I rested my body. It's up to me to do that, doctor. After that tomorrow I will repeat the same activity before....” R6

{Quote 25]“........If I'm really anxious, it will interfere my studies. So it's hard for me to learn. When I was worried, I couldn't go to study. I calm down first then study. But when I succed calm down with myself, it finally becomes my spirit....” R13

Based on the data mentioned here, it can be said that 10 determinant factors are directly related to learning behavior. These determinant factors can be categorized into two factors, namely, internal factors and external factors. Internal factors consist of task values, goal orientation, weakness, impact failure, benefits of passing the examination, self-efficacy, beliefs, barriers to learning, and test anxiety. Meanwhile, the environmental factors as the external factors also determine learning behavior. All determinant factors are directly related to both qualities of learning and regulation of learning. But the impact of failure is only related directly to the regulation of learning, and barriers to learning are only related directly related to the quality of learning.

Discussion

This study identified the determinant factors that encourage students to learn while preparing for a high-stakes assessment. These factors consist of two major themes, namely, internal factors, which occur at an individual level, and external factors, which occur at interpersonal and community levels. Internal factors include task values, goal orientation, weakness, impact failure, benefits of passing the examination, self-efficacy, spiritual beliefs, barriers to learn, and anxiety. Meanwhile, the external factors are environmental factors that come from institutional regulations and peer influence.

These internal factors occur at the individual level and shape student motivation and consist of the perception of weakness, barrier, and the consequences of the task (impact failure and benefits of passing the examination). These perceptions make students have various emotional feelings. Feeling nervous, anxiety, or stress before the examination is called test anxiety (Messineo et al., 2015).

The results of this study were in line with those by Cilliers et al. (2010, 2012a) who wrote about the impact of summative assessments on student learning behavior, which consist of the process of cognitive and metacognitive activity, with the perception factor as one of the mechanisms of impact of assessment (Cilliers et al., 2010, 2012a). The perception factors that occur at the individual level include perceptions related to weakness (quotations 6 and 7), the perception of the impact of failure (quotations 8 and 9), the perception of benefits of passing the examination (quotation 10), and perceptions of barriers to learn (quotation 17).

This perception factor is a determining factor in student learning behavior before the examination at the individual/intrapersonal level. This perception factor also clarifies and supports the research of Al Kadri et al. (2011), which states that student behavior in studying before the examination is largely determined by their perception of the examination to be taken (Al Kadri et al., 2009, 2011). With the perception of the threat of failure and the expectancy of passing the examination at the individual level, it can cause feelings ranging from nervousness to anxiety before the examination. Feelings of anxiety before this examination can encourage students to make changes in learning behavior by studying seriously (quotation 23). This is in line with the research by Messineo et al. (2015), who found that anxiety before examinations can mediate learning behavior (Messineo et al., 2015). But, this research found that not all of the anxiety will drive learn because some students will stop learning for 1 day for relaxing when they feel high anxiety, as stated by respondents R6 and R13 (quotations 24 and 25). Thus, this anxiety condition is still an interesting topic for further research to discuss about a positive or negative effect on student learning behavior.

Different perceptions in responding to the learning environment also occur in this study as happened to R7 (quote 17) and R13 (quote 18). Some respondents saw the pandemic condition with online learning to be an obstacle in learning, but there were also respondents who saw it as an advantage. In this perception factor, there are students who saw the COVID-19 pandemic as not an obstacle in learning, but on the contrary, they are happy with online learning (quotation 18). R13 said he preferred the COVID-19 pandemic because it was enough to study in the room without having to go to campus and without bringing books. The perspective in seeing the conditions of the surrounding environment is different for each respondent. Accordingly, this perception factor really influences student behavior in learning.

In addition to the perception factor, there are also an expectancy component (viz., self-efficacy, task value, and goal orientation) and a value component (viz., spiritual beliefs and normative beliefs) that students have. There is also an affective component that occurs, such as test anxiety. The expectancy, value, and affective components are the shapers of student motivation (Pintrich and De Groot, 1990; Pelaccia and Viau, 2017; Rosenzweig et al., 2019).

Another internal factor is the same with the value component in the previous study (Pintrich and De Groot, 1990) that found beliefs, namely, normative belief and spiritual belief. This normative belief greatly influences student learning behavior before the examination (quotation 16) including the belief that if they seriously study the task type to be tested, they will be successful. This belief occurs because of cues from the social environment, which can be from friends or mentors. This finding is in line with previous research conducted by Cilliers et al., (2010, 2012a), indicating that normative belief was included in contextual factors, as found in the research of Moghaddam et al. (2019) (Cilliers et al., 2010, 2012a). Beliefs in these norms are able to encourage students to learn in order to be successful and pass the examination.

However, this study found determinant factors that have not been discussed in previous studies, such as those related to spiritual beliefs, which are the mechanism factors for learning. This spiritual belief is a belief that originates in self-guiding God who determines everything that happens in this world, including the existence of life and death (Gall et al., 2011; Watkins et al., 2013). This spiritual belief is very prominent for students in determining learning resources. They only choose official or learning resources permitted by the owner. They feel uncomfortable when choosing unpermitted learning resources because they feel that they are not blessed, like in quote 14. This spiritual belief always guides student learning activities starting from the learning process and regulation activity of learning, which can be seen in quotation no. 15. In choosing learning resources, the students have to ask permission from the owner for reasons of goodness and blessing from God. If they do not get permission or do it without permission from the owner of the learning resources, there is a kind of fear related to blessings and benefits in future. In addition to behavioral changes in the student learning process, it turns out that students also carry out activities that are completely unrelated to academic matters, namely, worship activities. They engage in worship activities so that they are given ease, fluency, and success in their examinations, which can be seen in quote 15. Accordingly, this spiritual belief is a driving force for changes in students' learning and non-learning behavior, whose ultimate goal is success in the examination. This spiritual belief is a mechanism factor that needs to be explored further to see its effect on learning behavior, especially on success in high-stakes assessments.

The external factors that consist of institutional regulation and peer influence as drivers of the learning behavior in this study are in line with the level of influence from an ecological perspective, or multiple levels of influence, as discussed by McLeroy et al. (1988) cited by Glanz and Rimer (2005). They wrote that there are five factors that influence a health behavior, namely, (1) intrapersonal or individual factors, (2) interpersonal factors, (3) institutional or organizational factors, (4) community factors, and (5) public policy factors (Glanz and Rimer, 2005). This present study found the levels of influence, which become the external factors that occur at the interpersonal and community levels (institutional factors and community factors).

Another determinant factor is the mechanism factor that occurs at the interpersonal (quotations 20 and 21) and community (quotation 22) levels. At the interpersonal level, the determinant factor is related to peer influence. Peers will influence student learning behavior in attending tutoring outside the campus as well as group or independent study. Students following a tutoring or group study are very dependent on their peers (quotation 20). There are students who initially do not want to take tutoring outside campus for learning, but they are asked or influenced by their friends so that they become involved in certain tutoring activities for learning. There are also students who initially did not want to learn, but their friends invited them to study, and they also study together (quotation 19). This pattern shows that the influence at the interpersonal level also determines a change in behavior. Meanwhile, the community level also greatly influences student learning behavior, which consists of institutional factors and community factors. This institutional factor is related to institutional policy in organizing the learning process before the examination. There are institutions that require students to take tutoring on campus before the examination (quotation 22) so that students will try to take part in learning activities that have been determined by their respective institutions or engage in pre-scheduled study groups to prepare for the examination. Accordingly, at the community level, this becomes an effective way to be able to mediate changes in student learning behavior. While the community factors consist of study groups inside or outside the campus such as tutoring or independent study groups, the existence of this study group can also be effective in encouraging students to study individually. As a result, the behavior of students by creating a study group or attending tutoring either formal or informal also has an impact on student motivation to learn. By being in a good learning community, they can maintain consistency in learning in good conditions as well. The habit of students who prefer to gather to study together and provide assistance to one another is strongly influenced by local cultural conditions. In Indonesia, this behavior in Javanese language is known as “guyub”, which in the Big Indonesian Dictionary (KBBI) has the meaning “rukun” and berguyub, which means gather and group.

These factors that occur at the interpersonal and community levels, when viewed in previous research, are included in contextual factors, such as peer pressure, organizational regulation, society norms, and expectations. However, their relationship with learning behavior has not been explained in a straightforward manner (Moghaddam et al., 2019). The study only explored the existence of four interrelated categories, namely, assessment programs, mechanism factors, contextual factors, and impacts on learning behavior. There is a different determination factor in this study compared to previous research. This factor serves as a similar mechanism of learning behavior change according to the results of research conducted by Al Kadri et al. (2011). The results show cultural and emotional differences can have an impact on differences in student perceptions in viewing assessments so that there are differences in learning styles (Al Kadri et al., 2011).

The student behavior in preparing for the national licensing examination is strongly influenced by internal factors at the individual level and external factors at the interpersonal and community levels. These findings are also in line with the ecological perspective, which explains change requires simultaneous interaction of the influence of each level, namely, the intrapersonal, interpersonal, and community levels (Glanz and Rimer, 2005). The influence of each level in changing students' learning behavior should be designed to produce the expected changes in learning behavior. Accordingly, this research has succeeded in exploring the determinant factors that become the mechanisms of an assessment that can encourage changes in student learning behavior in the Indonesia context.

Limitations

The limitations in this study include the methodology of collecting data using in-depth interviews. In-depth interviews were conducted as online interviews with half of the participants, and during the online interviews, there could be signal interference so that noise/interference in communication could occur, but it was minimized by further communication via WhatsApp and playing back the recording for member checking. The other aspects were related to the respondents' misperceptions in answering the researcher's questions related to the impact of the assessment on their learning behavior. The respondents may experience misinterpretation in answering questions related to previous knowledge and experience (Cohen et al., 2002). However, the interview process was conducted carefully by paying attention to the responses of the respondents in answering questions that had been prepared as a guide. Data saturation was obtained during the interviews and data analysis, so the researchers believe that the data obtained are good in quality. The other limitation was related to the participants who did not provide feedback on the final findings.

The strength of this research is in the number of participants who were selected based on the maximum variation of sampling, with almost the same proportion of men and women, represented by a variety of academic abilities at a public faculty of medicine with an A accreditation level and a private faculty of medicine with a B accreditation level. In addition, the participants were also added from regional origins that are geographically and culturally diverse, in order to better apply the theoretical coding. Also, this study was conducted during the COVID-19 pandemic, which gave a different set of challenges to the situation in the learning environment.

Conclusion

This research found the determinant factors of learning that included internal factors at the individual level and external factors at the community level that drive students' learning before they take a high-stakes assessment. Student behavior in preparing for the national licensing examination is strongly influenced by internal factors at the individual level and external factors at the interpersonal and community levels. Internal factors include task value, goal orientation, weakness, benefits of passing the examination, self-efficacy, beliefs, and test anxiety. Meanwhile, the external factor is the environmental factor that consists of peer influence and institutional regulation. These factors determine the medical student learning behavior in the national medical competency examination during the COVID-19 pandemic in Indonesia, both in quality and regulation of learning. Spiritual factors were found as the determinant factors in this study, which were not found in the previous studies. Spiritual factors and environmental factors determine all of the learning behaviors, which consist of quality of learning and regulation of learning. Meanwhile, the impact of failure only determines the regulation of learning, while the barrier to learning only determines the quality of learning.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by Medical and Health Research Ethics Committee (MHREC) of the FKKMK UGM—Dr. Sardjito General Hospital, Indonesia, with file number KE/FK/0779/EC/2020. The patients/participants provided their written informed consent to participate in this study.

Author contributions

MF conceived and designed the analysis, collected the data, contributed data or analysis tools, performed the analysis, and wrote the paper. YS conceived and designed the analysis, contributed data or analysis tools, and review and revise the paper. GR conceived and designed the analysis, contributed data or analysis tools, review the data analysis, and review and revise the paper. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by Universitas Gadjah Mada Program Rekognisi Tugas Akhir Tahun 2021 [Grant Number 3190/UN1/DITLIT/DIT-LIT/PT/2021].

Acknowledgments

The authors wish to thank all participating medical students from the FK Unisma, FK-KMK UGM, and East Indonesia Medical Faculty. We also thank all our colleagues, who helped as coders. We also thank transcriptDOC, who helped as transcribers.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Al Kadri, H. M., Al-Moamary, M. S., Magzoub, M. E., Roberts, C., and Van der Vleuten, C. P. M. (2011). Students' perceptions of the impact of assessment on approaches to learning: a comparison between two medical schools with similar Curricula. Int. J. Med. Educ. 2, 44–52. doi: 10.5116/ijme.4ddb.fc11

CrossRef Full Text | Google Scholar

Al Kadri, H. M., Al-Moamary, M. S., and Van der Vleuten, C. P. M. (2009). Students' and teachers' perceptions of clinical assessment program: a qualitative study in a PBL curriculum. BMC Res. Notes 2, 263. doi: 10.1186/1756-0500-2-263

PubMed Abstract | CrossRef Full Text | Google Scholar

Al-Kadri, H. M., Al-Moamary, M. S., Roberts, C., and van der Vleuten, C. P. M. (2012). Exploring assessment factors contributing to students' study strategies: literature review. Med. Teach. 34, S42–S50. doi: 10.3109/0142159X.2012.656756

PubMed Abstract | CrossRef Full Text | Google Scholar

Archer, J., Lynn, N., Roberts, M., Gale, T., and Bere, S, R. (2016). The medical licensing examination debate. Reg. Gov. 11, 315–322. doi: 10.1111/rego.12118

CrossRef Full Text | Google Scholar

Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., and Van der Vleuten, C. P. M. (2006). The wheel of competency assessment: presenting quality criteria for competency assessment programs. Stud. Educ. Eval. 32, 153–170. doi: 10.1016/j.stueduc.2006.04.006

CrossRef Full Text | Google Scholar

Broekkamp, H., and Van Hout-Wolters, B. H. A. M. (2007). Students' adaptation of study strategies when preparing for classroom tests. Educ. Psychol. Rev. 19, 401. doi: 10.1007/s10648-006-9025-0

CrossRef Full Text | Google Scholar

Cilliers, F. J. (2015). Is assessment good for learning or learning good for assessment? A. Both? B. Neither? C. It depends? Perspect. Med. Educ. 4, 280–281. doi: 10.1007/s40037-015-0229-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Cilliers, F. J., Schuwirth, L. W. T., Adendorff, H. J., Herman, N., and Van der Vleuten, C. P. M. (2010). The mechanism of impact of summative assessment on medical students' learning. Adv. Heal. Sci. Educ. 15, 695–715. doi: 10.1007/s10459-010-9232-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Cilliers, F. J., Schuwirth, L. W. T., Herman, N., Adendorff, H. J., and van der Vleuten, C. P. M. (2012a). A model of the pre-assessment learning effects of summative assessment in medical education. Adv. Heal. Sci. Educ. Theor. Pract. 17, 39–53. doi: 10.1007/s10459-011-9292-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Cilliers, F. J., Schuwirth, L. W. T., and Van der Vleuten, C. P. M. (2012b). Modelling the pre-assessment learning effects of assessment: evidence in the validity chain. Med. Educ. 46, 1087–1098. doi: 10.1111/j.1365-2923.2012.04334.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Cilliers, F. J., Schuwirth, L. W. T., and van der Vleuten, C. P. M. (2012c). A model of the pre-assessment learning effects of assessment is operational in an undergraduate clinical context. BMC Med. Educ. 12, 1–9. doi: 10.1186/1472-6920-12-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Cohen, L., Manion, L., and Morrison, K. (2002). “Ex post facto research,” in Research Methods in Education, eds L. Cohen, L. Manion, and K. Morrison (New York, NY: Routledge).

PubMed Abstract | Google Scholar

Corbin, J., and Strauss, A. (2008). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Thousand Oaks, CA: Sage Publications, Inc.

PubMed Abstract | Google Scholar

Dillon, G. F., Swanson, D. B., McClintock, J. C., and Gravlee, G. P. (2013). The Relationship Between the American Board of Anesthesiology Part 1 Certification Examination and the United States Medical Licensing Examination. J. Grad. Med. Educ. 5, 276–283.

PubMed Abstract | Google Scholar

Firmansyah, M. (2015). Studi kualitatif dampak uji kompetensi dokter Indonesia terhadap pembelajaran pada mahasiswa kedokteran [Study on the impact quality of Indonesian doctor competency test on learning in medical students]. Indonesian J. Med. Educ. 4, 129–139. doi: 10.22146/jpki.25285

CrossRef Full Text | Google Scholar

Gall, T. L., Malette, J., and Guirguis-Younger, M. (2011). Spirituality and religiousness: a diversity of definitions. J. Spiritual. Ment. Heal. 13, 158–181. doi: 10.1080/19349637.2011.593404

CrossRef Full Text | Google Scholar

Glanz, K., and Rimer, B. K. (2005). Theory at a Glance: A Guide for Health Promotion Practice. Bethesda, MD: U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health, National Cancer Institute.

Google Scholar

Knight, P. (2001). A Briefing on Key Concepts: Formative and Summative, Criterion and Norm-Referenced Assessment, 7th Edn. York: Learning and Teaching Support Network.

Google Scholar

Leung, S. F., Mok, E., and Wong, D. (2008). The impact of assessment methods on the learning of nursing students. Nurse Educ. Today 28, 711–719. doi: 10.1016/j.nedt.2007.11.004

PubMed Abstract | CrossRef Full Text | Google Scholar

McLeroy, K.R., Bibeau, D., Steckler, A., and Glanz, K. (1988). An Ecological Perspective on Health Promotion Programs. Heal. Educ. Behav. 15, 351–377.

PubMed Abstract | Google Scholar

Melnick, D. E. (2009). Licensing examinations in North America: Is external audit valuable? Med Teach. 31, 212–214. doi: 10.1080/01421590902741163

PubMed Abstract | CrossRef Full Text | Google Scholar

Messineo, L., Gentile, M., and Allegra, M. (2015). Test-enhanced learning: Analysis of an experience with undergraduate nursing students. BMC Med. Educ. 15, 1–7. doi: 10.1186/s12909-015-0464-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Moghaddam, A. K., Khankeh, H. R., Shariati, M., Norcini, J., and Jalili, M. (2019). Educational impact of assessment on medical students' learning at Tehran University of Medical Sciences: a qualitative study. BMJ Open 9, e031014. doi: 10.1136/bmjopen-2019-031014

PubMed Abstract | CrossRef Full Text | Google Scholar

Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med. Teach. 33, 206–214. doi: 10.3109/0142159X.2011.551559

PubMed Abstract | CrossRef Full Text | Google Scholar

Norcini, J., Anderson, M. B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2018). 2018 Consensus framework for good assessment. Med. Teach. 40, 1102–1109. doi: 10.1080/0142159X.2018.1500016

PubMed Abstract | CrossRef Full Text | Google Scholar

Norman, G., Neville, A., Blake, J. M., and Mueller, B. (2010). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Med. Teach. 32, 496–499. doi: 10.3109/0142159X.2010.486063

PubMed Abstract | CrossRef Full Text | Google Scholar

Pelaccia, T., and Viau, R. (2017). Motivation in medical education. Med. Teach. 39, 136–140. doi: 10.1080/0142159X.2016.1248924

PubMed Abstract | CrossRef Full Text | Google Scholar

Pintrich, P. R., and De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. J. Educ. Psychol. 82, 33–40. doi: 10.1037/0022-0663.82.1.33

CrossRef Full Text | Google Scholar

Rosenzweig, E. Q., Wigfield, A., and Eccles, J. S. (2019). Expectancy-value theory and its relevance for student motivation and learning. Cambridge Handb. Motiv. Learn. 617–644.

Google Scholar

Schuwirth, L., and Van der Vleuten, C. (2004a). Merging views on assessment. Med. Educ. 38(12), 1208–1210. doi: 10.1111/j.1365-2929.2004.02055.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Schuwirth, L. W. T., and Van der Vleuten, C. P. M. (2004b). Different written assessment methods: what can be said about their strengths and weaknesses? Med. Educ. 38, 974–979. doi: 10.1111/j.1365-2929.2004.01916.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Schuwirth, L. W. T., and Van der Vleuten, C. P. M. (2011). Programmatic assessment: from assessment of learning to assessment for learning. Med. Teach. 33, 478–485. doi: 10.3109/0142159X.2011.565828

PubMed Abstract | CrossRef Full Text | Google Scholar

Shumway, J. M., and Harden, R. M. (2003). AMEE Guide No. 25: the assessment of learning outcomes for the competent and reflective physician. Med. Teach. 25, 569–584. doi: 10.1080/0142159032000151907

PubMed Abstract | CrossRef Full Text | Google Scholar

Van der Vleuten, C. P. M., and Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Med. Educ. 39, 309–317. doi: 10.1111/j.1365-2929.2005.02094.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Watkins, Y. J., Quinn, L. T., Ruggiero, L., Quinn, M. T., and Choi, Y. K. (2013). Spiritual and religious beliefs and practices and social support's relationship to diabetes self-care activities in African Americans. Diabetes Educ. 39, 231–239. doi: 10.1177/0145721713475843

PubMed Abstract | CrossRef Full Text | Google Scholar

Appendix 1

In-depth interview guideline

This interview was conducted to explore the preparation of students before the exam.

Questions to explore the impact of exams on student learning with the probing guideline if participants did not explain the topic that we need.

1. About exams and exam types.

1.1 What does UKMPPD mean to you?

1.2 How do you feel now when you take UKMPPD?

1.3 What do you do to deal with these feelings?

2. Questions to explore the process of self-reflection and targeting learning outcomes.

2.1 What benefits can you get from the UKMPPD exam?

2.2 What is your goal in joining UKMPPD?

2.3 Is there a target value?

2.4 Is it just taking a test and getting a mediocre score?

2.5 Is there any particular motivation?

3. Study strategy (before the exam).

3.1 What preparations did you do before joining the UKMPPD?

3.2 How do you learn in dealing with UKMPPD?

3.3 Is there a difference between studying during college and when preparing for UKMPPD?

3.4 When did you start preparing for UKMPPD?

3.5 Are there any obstacles while studying during this pandemic?

3.6 What is your extraordinary effort in dealing with UKMPPD?

3.7 Is there time off to study?

3.8 Additional efforts that you are currently taking to prepare for UKMPPD (e.g. a kind of additional course/guidance for UKMPPD before the actual exam) Explain!

3.9 How much additional time did you spend preparing for this exam beyond the scheduled study time?

3.10 What are the average hours of study per day/per week to prepare for UKMPPD?

3.11 What learning resources do you use to prepare for UKMPPD/exams (certain books, clinical lecturers, mentors, people, etc.)?

3.12 How do you choose the material to study for the preparation of UKMPPD and are there certain/special topics that are a priority?

3.13 Where have you studied UKMPPD?

3.14 How do you manage your study time/time management for UKMPPD preparation?

3.15 When did you start preparing for UKMPPD specifically?

3.16 Have you made any efforts to stay consistent in learning ahead of UKMPPD?

3.17 Are there problems in your learning process? do you often forget? efforts to study effectively?

3.18 Did you feel confident before UKMPPD that you could do UKMPPD well? Reason?

Closing sentence

Thank you all for your willingness to participate in this interview. Please be willing to be contacted again if something is needed, especially for member checking.