Bridging the Gap: Using Consensus to Explore Entrustment Decisions and Feedback Receptivity in Competency-Based Emergency Medicine Residency Programs Through the Construction of a Q-Sample Incorporating a Delphi Technique

Background: Recent changes in medical education calls for a shift toward student-centered learning. Therefore, it is imperative that clinical educators transparently assess the work-readiness of their medical residents through entrustment-based supervision decisions toward independent practice. Similarly, it is critical that medical residents are vocal about the quality of supervision and feedback they receive. This study aimed to explore the factors that influence entrustment-based supervision decisions and feedback receptivity by establishing a general consensus among Taiwanese clinical educators and medical residents regarding entrustment decisions and feedback uptake, respectively.

Methods: In Q-methodology studies, a set of opinion statement (i.e., the Q-sample) is generated to represent the phenomenon of interest. To explore the factors that influence entrustment-based supervision decisions and feedback receptivity, a Q-sample was developed using a four-step approach: (1) literature search using electronic databases, such as PubMed and Google Scholar, and interviews with emergency clinical educators and medical residents to generate opinion statements, (2) thematic analysis and grouping using The Model of Trust, the Ready, Wiling, and Able model, and the theory of self-regulated learning, (3) translation, and (4) application of a Delphi technique, including two expert panels comprised of clinical educators and medical residents, to establish a consensus of the statements and validation for a subsequent Q-study.

Results: A total of 585 and 1,039 statements from the literature search and interviews were extracted to populate the sample of statements (i.e., the concourse) regarding entrustment-based supervision decisions for clinical educators and feedback receptivity emergency medicine residents, respectively. Two expert panels were invited to participate in a Delphi Technique, comprised of 11 clinical educators and 13 medical residents. After two-rounds of a Delphi technique, the panel of clinical educators agreed on 54 statements on factors that influence entrustment-based supervision decisions and were categorized into five themes defined by the Model of Trust. Similarly, a total of 60 statements on the factors that influence feedback receptivity were retained by the panel of medical residents and were categorized into five themes defined by the Ready, Willing, and Able model and the theory of self-regulated learning.

Conclusion: Though not exhaustive, the key factors agreed upon by clinical educators and medical residents reflect the characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. This study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. Additionally, incorporation of the Delphi technique further adds to the existing literature and puts emphasis as an important tool that can be used in medical education to rigorously validate Q-statements and develop Q-samples in various specialties.

Introduction

A fundamental aspect of competency-based residency training is the decision to entrust a medical resident with the responsibility to care for a patient with minimal supervision (1). When made negligently, the decision can compromise patient safety. Clinical educators (CEs) rely on multiple sources of information to provide them with knowledge regarding how much supervision a medical resident may need during clinical training (2, 3). Entrustment decisions regarding supervision requires not only for CEs to reflect on their decision making process, but also for medical residents to reflect on their own skillset and characteristics (4, 5). Evidence suggest that the multidimensional nature of entrustment decisions are largely subjective and are both task dependent and task independent (3, 6–8).

To date, only a few studies have used a Delphi technique to build a consensus regarding how experienced CEs make entrustment-based supervision decisions in clinical practice (9, 10). Although these studies have established a consensus and shared mindset on factors that influence entrustment decisions among CEs (9, 10), they solely focus on the aspects related to a medical residents' characteristics without including that of CEs. Literature has shown, however, that these decisions can be influenced by the nature of the task and a supervisors' individual characteristics (6). Therefore, further research is needed to establish a consensus on the path to entrustment while taking into consideration the nature of the task and a CEs individual attributes (11). To contribute to the generalizability of factors considered to be important when making entrustment decisions, research in various countries should be conducted (9, 12). Results from such research can be used to establish standard of practice in regards to supervision decisions made by CEs in the clinical workplace (13).

Supervision facilitates learning in the clinical workplace and requires guided interactions between CEs, medical residents, and patients, giving medical residents exposure to hands-on practice (14, 15). The process of supervision allows CEs to provide a foundation for medical residents as they engage in direct practice (15) while the process of feedback empowers medical residents with the necessary knowledge to improve their clinical and procedural skills (15, 16). Feedback in the clinical setting manifests in many forms, it can be either explicit—given after assessment or focused on overall performance (i.e., written and verbal), or implicit (i.e., body responses and reactions) (17). Research has shown that entrustment decisions made after evaluating a medical resident's behaviors are, therefore, a reflection of their displayed level of competency (18). As such, the level of autonomy assigned to medical residents based on entrustment decisions is perceived as a form of implicit feedback regarding their entrustable practices (17–19). A medical resident's perception of the level of autonomy given to them during clinical practice can affect how they view their own clinical competency and influence how self-evaluative judgment skills regarding their clinical practice is developed (18–20).

However, formative feedback alone does not always yield the expected impact on learners (21). Studies note a gap between feedback received and how it is implemented, highlighting the role recipients need to play as active synthesizers and evaluators of feedback in order for learning to take place (16, 22). Several factors have been noted to influence the credibility and reliability of feedback received, e.g., the delivery method (15, 16, 23). Previous research has emphasized the development of feedback delivery skills among CEs (16, 22). Though, recent studies suggest a shift in research to focus on incorporating a learner's perspective of feedback credibility and reliability is necessary to bridge the gap between feedback and its intended outcome (24, 25). Through these findings, we propose that a learner's evaluation of feedback is likely to bring new perspectives through establishing a consensus on the factors that influence feedback receptivity. Insight on how feedback is received by medical residents can be used to enhance CEs feedback delivery skills (23, 24).

This study is part of a larger multi-year study regarding the incorporation of competency-based medical education and its subsequent assessment methods in emergency medicine in Taiwan. Recently, Taiwan was named as an international hub for Accreditation Council for Graduate Medical Education regional faculty development (26). We consider the path to entrustment decisions and feedback receptivity as two separate but crossing paths. Therefore, this study aims to establish a consensus on factors that affect a CEs path to making entrustment-based supervision decisions and the factors that affect a medical residents' receptivity to feedback received. A concourse of statements regarding such topics of interest was gathered from literature and semi-structured interviews. Additionally, we used a Delphi technique to establish expert consensus on critical factors that influence both entrustment decisions among CEs and medical residents' feedback receptivity.

Methods

Setting and Participants

We conducted this study in Taiwan from March 2020 to June 2021. Emergency medicine was among the first specialties to implement competency-based residency training programs amid the recent medical education reform in Taiwan (27, 28). In competency-based residency training programs, all senior physicians are expected to take part in the training of residents and medical residents are required to actively engage in their own training by seeking feedback and reflecting on their own progress (27, 29). Therefore, participants of our study included clinical educators and medical residents of various specialties practicing medicine in Taiwan. Participants were recruited via snowball sampling, with initial contact made by the principal investigator to the participants.

Q-Methodology and a Delphi Technique

Increasing in popularity, Q-methodology is a mixed-methods approach to exploring human subjectivity (30–32). The goal of a Q-methodology study is to reveal different patterns of thoughts and perspectives of participants through ranking and sorting statements on a continuum of meaningfulness (31, 33). It draws on both qualitative and quantitative methodologies for data collection and analysis. However, there remain concerns about the best practice for Q-sample development, limited guidance on constructing a Q-sample, and methods to measure the validity and reliability of a Q-sample (33, 34).

The Delphi technique is a group facilitative method used with the intention of developing an expert-based judgment on a chosen topic to reach group consensus involving panel members who are experts in a selected field. Panelists remain anonymous from one another to reduce dominant personalities influencing the consensus process by one or more experts (35, 36). While there is no consensus for the exact methodology for conducting a Delphi study in literature, it can be argued that a typical Delphi technique is comprised of five steps: (1) a problem is presented to a panel of experts, (2) panel members individually respond through a structured questionnaire or interview, (3) data is gathered, analyzed, and reworked toward collective agreement, (4) repetition of steps two and three for several rounds, as needed, and (5) collective agreement is achieved through statistical analysis (37). After each round of the Delphi technique, feedback is provided by the panel members. Thereafter, panel members are given the opportunity to discuss with the research team and change their opinions in subsequent rounds. The entire iterative process occurs until a consensus is reached.

Constructing the Q-Sample for the Delphi Technique

Similar to the framework set by Kirschbaum et al. (38), construction of the Q-samples consisted of four steps and four points of refinement (Figure 1).

FIGURE 1

Figure 1. Four-step method for Q-sample construction.

Step 1: Gathering Initial Thoughts—Literature Search and Interviews

Defining the concourse is the first step of a Q-methodology study. The concourse is a collection of possible statements that represent opinions of a given topic (39, 40). Using both ready-made (i.e., statements gathered from print media such as newspaper articles, magazines, and scientific literature) and naturalistic concourses (i.e., statements gathered from participants through questionnaires, interviews, and focus groups) (39, 41), opinion statements were gathered through a literature review and semi-structured interviews. Incorporating the use of scientific literature brought forth broader, cross-specialty perspectives regarding issues of entrustment decisions, clinical supervision, and feedback receptivity. Additionally, it facilitated the use of a Delphi technique for consensus by engaging CEs and medical residents from various specialties. Simultaneously, interviews with emergency CEs and EMRs was crucial to obtain specialty-specific viewpoints about clinical teaching, supervision, and learning through feedback in the emergency department and further guided the literature search. Material for the concourses aim to represent and capture the depth of existing opinions on the topic at hand-entrustment decisions and feedback receptivity (33, 40, 42).

Scientific Literature

A systematic search using PubMed and Google Scholar was independently conducted by three authors (NSN, RSC, and MK) from inception to April 2021. We used the following search terms for CEs: (clinical educator OR medical teacher OR supervisor OR attending physician) AND (decisions OR views OR attitude OR perceptions) AND (entrustment OR supervision OR scaffolding OR independent practice OR autonomous practice OR progressive independence). Additionally, the following search terms for EMRs: (resident OR medical trainee OR post graduate medical student OR emergency medicine resident) AND (feedback OR feedback synthesis OR feedback evaluation OR feedback perception OR feedback appraisal) AND (credibility OR reliability OR acceptability OR receptivity). Back referencing was used to identify potential studies and relevant citations to be included in our analysis.

Articles published in English and at any time were retained for analysis. Literature retained were those that met the following criteria: research published with clear relevance to the subject of factors that influence entrustment decisions by supervisors on trainees, research on feedback in medical education or medical training, performed in any country, and research focusing on factors affecting feedback receptivity. Quotes, statements, and key phrases that represented an opinion of a CE or medical resident concerning supervision, entrustable professional activities, entrustment decisions, and interactions between supervisors and trainees in the clinical setting were extracted. Additional articles were identified from reference lists of included papers.

Semi-structured Interviews

Semi-structured, one-on-one interviews with 13 emergency CEs and 11 EMRs were conducted to extract opinion statements and to observe for recurring themes. Interviews were conducted virtually through Zoom, a video-telephony propriety software program, audio recorded, and transcribed verbatim. CEs were initially prompted to answer questions regarding difficulties faced when making supervision-based entrustment decisions (Appendix 1). On the other hand, EMRs were initially prompted with questions regarding their opinions on the assessment tools used in their residency training programs, including strengths, weaknesses, and the credibility of feedback received (Appendix 2). Participants were affiliated with 9 different hospitals, each with emergency medicine residency training programs approved by the Residency Review Committee through the Taiwan Ministry of Health and Welfare.

To be eligible, emergency CEs interviewed had at minimum Five years of clinical teaching experience and were program directors or in a leadership position, while EMRs were in their third or 4th year of residency training. Details of the participants are shown in Table 1. The eligibility criteria was established to ensure participants had experienced the transition from traditional workplace-based assessment to competency-based medical education evaluation. Participants provided written informed consent prior to the interview and verbal consent prior to the audio recording. The interviews lasted between 30 and 60 min. Among emergency CEs, the goal of the interviews was to explore factors influencing entrustment while among EMRs, the goal was to explore feedback receptivity and how these decisions influenced their professional development and educational process.

TABLE 1

Table 1. Demographics of interview participants (N = 24).

Step 2: Thematic Grouping and Theoretical Considerations

Once all search efforts were exhausted and interviews were completed, all extracted information were refined into cohesive phrases and then into opinion statements. Like statements and statements with similar language or meaning were combined and grouped categorically using theoretical frameworks for each target population. The theoretical frameworks were used as guides to ensure that all possible viewpoints were accurately represented.

Clinical Educators

Drawing from Roger Mayer and colleagues' Model of Trust (8), the conceptual framework of the entrustment decision making process aims to advance the understanding of how ad-hoc entrustment decisions are made in the day-to-day clinical practice (6) (Figure 2). The model illustrates the complexity of the entrustment decision making process in the clinical context, noting that these decisions are dependent on time, context, and the task at hand. Equally, these decisions are influenced by multiple factors related to the relationship fostered between CEs and medical residents (6, 7). Interactions among these factors determine the level of entrustment and supervision that a CE is likely to provide to a medical resident (6). Therefore, this model served as the foundation for thematic analysis of extracted statements derived from the aforementioned Step 1.

FIGURE 2

Figure 2. Model of trust adapted from the conceptual framework of the entrustment decision-making process (6).

The extracted statements were categorized into five categories: (1) relationship between a supervisor and trainee, (2) contextual factors, (3) supervisor characteristics, (4) task complexity, and (5) trainee characteristics (6, 8). Every statement was assigned to a single thematic category.

Medical Residents

Feedback is essential in learning, however there are many factors that can influence the uptake of feedback by the learner. Developed by Garino (23), the Ready, Willing, and Able (RWA) model is a theoretical framework that explains the successful use of feedback (Figure 3). The model stipulates that upon receiving feedback, a learner goes through a cycle of valuing the messenger, valuing the message, makes meaning from the message, and compares it to their own self-evaluation. Once a learner has completed the cycle, they can then choose to engage in and employ adaptive learning strategies or dismiss the feedback received (23).

FIGURE 3

Figure 3. Cycle of feedback valuing derived from the five thematic categories derived from the Ready, Willing, and Able model and self-regulated learning (23, 43).

The successful use of feedback through the RWA model is set upon the foundation of self-regulated learning (SRL). SRL refers to the adaptive strategies that enables learners to transform their mental abilities into academic performance (23, 43). Applications of metacognitive strategies can be described through three phases of performance, reflection, and forethought (43, 44). Through the phases, learners receive feedback and align it to their personal goals, monitor their learning through self-questioning, and reflect on their learning to track improvements. The theoretical frameworks set by the theory of SRL and the RWA model demonstrates five categories for feedback receptivity, including: (1) delivery method, (2) characteristics of the learner, (3) intention and motivation to providing feedback, (4) content of the feedback, and (5) feedback sender credibility. Each statement was assigned to a single thematic category.

In the subsequent steps, statements extracted for emergency CEs and EMRs were revised for concise language while checking for simplicity, use of personal pronouns, and applicability to the cultural context. Negatively expressed statements were avoided because any disagreement with these statements may introduce a negative connotation into the proceedings and invoke a strong response from the participant (45). For example, if a statement were framed “I do not prefer written feedback,” a participant may disagree with this (i.e., give it a negative ranking) to negate the negative (I do not) and create the positive (I do) (45). Therefore, to avoid confusion or bias toward our statements, where a negative connotation was present, the statement was reworded to maintain a neutral (e.g., “written feedback”) or positive viewpoint (e.g., “I prefer written feedback”). Remaining statements were then proofread for grammar, reworded for clarity, and made certain each statement described only a single issue.

Step 3: Statement Translation

Statements were translated from English to Chinese to ensure target populations could fully understand the statement meanings. Forward translation from English to Chinese was done independently by the principal investigator, who is a native Chinese speaker. Then, back-translation was collaboratively done by two members of the research team, who are proficient in Chinese. Once back-translated, translated statements were compared to the original to ensure the intended conceptual meaning was maintained during the translation process. Any variances in the translations were reviewed and resolved within a consensus discussion among the research team. After discussion, repeated statements, statements with similar meanings, and those deemed irrelevant to the study were removed.

Step 4: Application of a Delphi Technique

A Delphi technique was employed to achieve expert consensus on pivotal factors that influence both entrustment-based supervision decisions and feedback receptivity. A multidisciplinary team of CEs and medical residents were recruited purposively to participate on the expert panel. Experts were defined as CEs and medical residents, practicing and training in various specialties. CEs had, at minimum 5 years of teaching experience, while medical residents were in their 3rd or 4th year of residency training to ensure that participants had experienced the recent change from assessment-based learning to competency-based training. Opinions on the number of statements in a Q-sample still varies among theorists and researchers. However, McKeown and colleagues suggest that a Q-sample of 50-70 is adequate for most studies (46). Through a discussion among the research team, a Q-sample of ≤60 was deemed ample to reduce the possibility of oversaturation or repetitive statements in this study. The Delphi technique was conducted from May 2021 to June 2021. All rounds of the Delphi technique were conducted via offline, paper surveys that were emailed as an attachment to participant individually.

Given that emergency medicine is a complex dynamic specialty that requires specific training in many disciplines (47), panel members from various specialties were invited to participate in the Delphi technique. As part of their residency training, EMRs will rotate between various specialties outside of emergency medicine (47). Therefore, it is crucial that the viewpoints of CEs from other specialties are included as they provide EMRs with feedback about their clinical competencies. Similarly, medical residents from various specialties may rotate in emergency medicine as part of their training. However, research has shown a discrepancy in reliability of feedback provided to non-EMRs who during emergency medicine rotations (48). Therefore it is imperative to understand the perspective of medical residents from various specialties as well on the comprehensibility and receptivity of feedback provided.

A panel of 11 CEs practicing in 10 specialties, including pediatrics (1), internal medicine (1), emergency medicine (2), orthopedics (1), dermatology (1), nephrology (1), nursing (1), critical care (1), thoracic surgery (1), and respiratory therapy (1) were included in the expert panel. Additionally, an expert panel of 13 medical residents training in 7 differing specialties, including obstetrics and gynecology (2), dentistry (2), emergency medicine (2), internal medicine (1), Chinese medicine (2), pediatrics (2), and general surgery (2) were invited.

Round 1

Through purposeful sampling, the expert panel was selected based on individual expertise and knowledge as follows: (1) current experience, and (2) from different specialties to facilitate diversity of views and experience in an effort to reduce both researcher and specialty bias.11 CEs and 13 residents from four hospitals were recruited for participation. Details of the panel members are shown in Table 2. First, an explanation of the study and its purpose was provided to each participant. CEs were instructed to rank their opinions of the statements regarding factors that affect entrustment decisions while supervising medical residents. On the other hand, medical residents were instructed to rank their opinions of the statements in regards to their receptivity of feedback that influence their learning. Sequenced according to the five thematic groups defined by the Model of Trust and SRL/RWA, respectively, the panel members were asked to rank the statements on a 5-point Likert scale ranging from 1 (unimportant) to 5 (most important). Additionally, panelists were asked to leave a comment on why they ranked the statement as such and to provide suggestions for improving statement clarity, if applicable. Once completed, the median and interquartile range (IQR) for the continuous variables were calculated as recommended by Jones and Hunter (49). An IQR of ≤ 1 for a 5-point Likert scale is suggested as an acceptable indicator of consensus (50–52). In this study, statements that reached expert consensus (i.e., median 5 and IQR ≤ 1) were deemed conclusive and were not reevaluated in subsequent rounds. Furthermore, to ensure that our final Q-samples were ≤ 60 statements, statements that achieved an IQR ≤ 1 and a median of 4 were reevaluated by panel members in Round 2.

TABLE 2

Table 2. Demographics of panel members in the Delphi technique (N = 24).

Round 2

Participants from the first round of the Delphi technique were re-invited to participate in the second round to reevaluate statements that had reached consensus (i.e., IQR ≤ 1) but had a median of 4 to reduce the possibility of statement oversaturation. At the start, panelists were reminded of the study aim and explained the purpose of the second round (i.e., to reduce the number of statements and to obtain expert consensus). Median and IQR were once again calculated for each of the remaining statements to identify those that have reached expert consensus. Statements that reached expert consensus were revised as necessary based on feedback provided. Once Q-sample populations reached ≤ 60, no further rounds were necessary.

Results

Concourse Development and Statement Generation

In the process of identifying our concourse, a total of 585 and 1,039 statements from aforementioned interviews and scientific literature were extracted to populate the concourse for emergency CEs and EMRs, respectively, Figure 4 illustrates our concourse development and statement generation process. Of the 585 statements extracted for emergency CEs, 95 statements were derived from interviews with emergency CEs while 490 statements were derived from literature. After thematic sorting using the entrustment framework defined by the Model of Trust, 187 residual statements were retained. Similarly, of the 1,039 statements extracted for EMRs, 289 statements were gathered from interviews with EMRs and 750 statements were derived from literature. Then, 110 residual statements were combined and sorted thematically according to the five categories outlined by the SRL and RWA conceptual frameworks. Repeated statements and statements that did not reflect our research aims were removed. Upon translation of the statements, 130 statements and 99 statements were retained for CEs and EMRs, respectively.

FIGURE 4

Figure 4. Results of statement selection and reduction.

Building Consensus Through Delphi Technique

Round 1

From 130, a total of 75 statements achieved expert consensus among CEs. However, after a review of comments and suggestions from the 11 panelists, 6 statements were removed due to irrelevance or language ambiguity resulting in 69 statements retained. Similarly, of the 99 original statements, a total of 68 statements achieved expert consensus by the 13 medical residents However after evaluating comments from the panelists, one statement was deleted for ambiguity, leaving a total of 67 statements retained.

Round 2

A second round of a Delphi technique was employed to reevaluate important statements that did not achieve expert consensus among CEs and medical residents. Panel members from the first round of the Delphi technique were re-invited to reevaluate the remaining statements. The additional round of the Delphi allowed us to further reduce the number of statements and to achieve the goal of ≤60 statements.

Clinical Educators

Of the 75 statements from the first round, 69 statements achieved expert consensus, however 45 statements had reached median 4 and IQR ≤ 1 and were reevaluated in Round 2. Upon reevaluation, 30 statements achieved expert consensus. Therefore, after two rounds of a Delphi technique, panel members agreed on 54 statements (i.e., 24 statements from Round 1 and 30 statements from Round 2) on factors that influence entrustment-based supervision decisions (Table 3). Common factors mentioned in literature and interviews included their frequency of interaction with resident, a resident's level of training or seniority, and a resident's awareness of their skills accumulation and personal limitations to deliver safe patient care. Additionally, statements that were deemed context specific (i.e., statements that were repeatedly mentioned during interviews but infrequently found in literature) included a resident's ability to communicate with difficult family members, CEs sense of responsibility to educate a resident, and CEs attitude toward resident training responsibilities.

TABLE 3

Table 3. Q-Sample statements (N = 54) mapped to the model of trust domains.

All 54 statements were successfully assigned into one of the five categories defined by the Model of Trust: 2 statements into relationship between a supervisor and trainee, 8 statements into contextual factors, 8 statements into supervisor characteristics, 18 statements into task complexity, and 18 statements into resident characteristics. Furthermore, a resident's capacity for clinical reasoning, the complexity of the clinical case, and frequency of interaction with a resident were among the most mentioned factors in literature and in interviews that heavily influenced entrustment-based supervision decisions.

Medicine Residents

Among medical residents, 67 statements achieved expert consensus, however, 31 statements had reached median 4 and IQR ≤ 1 and were reevaluated in Round 2. Of these 31 statements, 24 statements achieved expert consensus. Therefore, a total of 60 statements (i.e., 36 statements from Round 1 to 24 from Round 2) on the factors that influence residents' receptivity to feedback were retained after Round 2 (Table 4). Factors that enhanced feedback receptivity ranged from feedback that is delivered in a clear and easy to understand manner to feedback from a source that has a good understanding of the curriculum. Furthermore, there were no statements deemed contextually specific among medical residents. All statements extrapolated from interviews were commonly found in international literature used in this study.

TABLE 4

Table 4. Q-sample statements (N = 60) mapped to the SRL and RWA^* model domains.

All 60 statements were successfully classified into the five categories defined by SRL and RWA: 17 statements into Delivery, 4 statements into characteristics of the recipient, 11 statements into intention or motivation for providing feedback, 13 statements into the content of the feedback, and 15 statement into Sender credibility. Furthermore, feedback provided in a safe and confidential environment, detailed feedback based on milestone assessments, and timely feedback provided after direct observation were among the most mentioned factors in literature and in interviews that influence feedback receptivity.

Discussion

This present study aimed to understand factors that influence entrustment in supervision decisions and feedback receptivity by establishing a general consensus among CEs and medical residents, respectively. To our knowledge, this is the first study of its kind to explore these two paths within a competency-based learning context in emergency medicine through the simultaneous development of two Q-samples and a Delphi technique to obtain expert consensus. Through the use of consensus and the steps for constructing a Q-sample to explore entrustment-based supervision decisions and feedback receptivity, this study provided deeper insight into entrustment, supervision, and feedback receptivity in the clinical context.

Entrustment Decisions and Clinical Supervision

This study revealed a sample of consensus-based factors that affect CEs decision to entrust medical residents with a clinical tasks. It aids in determining of the degree of supervision that is necessary to provide medical residents with the skills necessary to progress toward independent practice. Our study revealed factors that ranged from the frequency of interaction with a medical resident and their fostered relationship to factors regarding the context and level of risk associated with the clinical task. Globally, competency based medical education is rapidly emerging as the prominent paradigm across workplace-based education (53). Competency and milestone-based learning is designed to improve teaching and assessment of learners among broad domains (54). An important aspect of competency-based medical education is the decision to entrust a medical resident to perform clinical procedures, which is a complex task that does not come with a binary solution. Rather, it lies on a continuum from medical residents requiring a high degree of supervision and direct observation of procedures performed to medical residents receiving minimal supervision (55).

Though the decision to entrust a medical resident with a patient is a complex and subjective process, our study revealed the non-linear path toward deciding how CEs decide to reduce the levels of supervision and to entrust medical residents to perform patient care independently. By understanding key factors that CEs deem as key indicators of competency exhibited by an medical resident that influence their entrustment decisions, CEs and medical residents establish shared mental models that informs what constitute as competency in the clinical setting (56, 57). This shared mental model may guide new medical residents' on behaviors worth emulating and to continue behaviors worth reinforcing (56, 57). This can empower medical residents to participate in their learning process and contributes to efforts of instilling a student-centered learning culture in the clinical workplace (58).

Our factor statements also include statements related to task or situation dependent domains and a CEs characteristics. Insight into how various factors influence entrustment decisions is necessary for CEs reflective practice. Establishing consensus on a subjective matter, such as entrustment decisions, may reveal discrepancies between the theoretical assumptions and the reality of factors that influence their decisions in practice (59). It can reveal how a CEs personal and educational background along with contextual factors that form the hidden curriculum can influence supervision, teaching, and learning (60, 61). Previous studies have illustrated how a CEs first impressions of a medical resident tend to influence their subsequent entrustment and supervisions decisions, suggesting that training is essential to helping supervisors pay attention to this form of bias (62, 63). We suggest that by uncovering this shared mindset among CEs, our study can contribute to the discourse of how educators can modify their entrustment and supervision behavior to reduce bias.

Feedback Comprehension and Receptivity

Under competency-based medical education, CEs are advised that learners receive timely, specific, constructive, and fair feedback to enhance their learning (64). The Delphi technique revealed factors that influence medical residents' synthesis of feedback and receptivity, ranging from the delivery method of the feedback to individual characteristics of the recipient and the credibility of the feedback deliverer. In line with findings from contemporary literature, our results describe the importance of using various feedback delivery methods, the content of the feedback, and the credibility of the feedback provider (23, 43). Therefore, these study results reflect an agreement among medical residents of factors that influence the comprehension, evaluation, and receptivity of feedback.

While feedback receptivity may be subjective in nature, our results reiterate factors that resonate most with medical residents across various specialties. Additionally, it bridges the gap between a theoretical understanding and the practical reality of what medical residents perceive useful feedback relative to other forms of feedback. Research has shown that without a common frame of reference and perceptions of quality feedback can be detrimental to a learner's engagement in their learning process and may prevent them from fully maximizing the benefits offered by formative feedback (65). Insights into medical residents' shared mindset on the relative importance of various factors that influence their receptivity to feedback is crucial in helping CEs develop the appropriate mechanism for giving feedback to residents.

Study Implications

Our study serves as a foundation toward the development of initiatives designed to help CEs optimize their supervision practices through the learners' perspective as receivers of feedback through entrustment-based supervision decisions. We suggest that these decisions and practices should be personalized to the learner so that they reflect individual skillsets, which is essential in providing transparency in assessment. Feedback should be informed by CEs and that the level of entrustment and supervision constitutes as a form of feedback, aligning with previous studies that bridge supervision and feedback (66). This indicates that the amount of supervision assigned to trainees is a form of implicit feedback (17–19). Finally, this study contributes to the clinical supervision and feedback literature by providing representative factors that influence entrustment-based supervision decisions and feedback receptivity. This serves as a foundation for fostering a shared mental model between CEs and medical residents across various specialties as the cornerstones of entrustment-based supervision decisions and applicable feedback.

To our knowledge, this is the first study of its kind in constructing a concourse for a Q-methodology study through incorporating a Delphi technique to explore factors that influence supervision entrustment decisions among emergency CEs and EMRs feedback receptivity. Apart from using the two types of concourses (i.e., naturalistic and ready-made), the strength of this study lies at the use of three conceptual models to inform the thematic analysis of the statements. Using established theoretical models structures the construction of a Q-sample and ensures that statements represent different facets of the Research Topic (33, 38, 45). Although the statements extracted primarily originated from international literature, a Delphi technique was incorporated in our study to obtain validation from Taiwanese experts for relevance to the broader East Asian healthcare setting. Through using theoretical frameworks and a Delphi technique, the resulting Q-samples have greater plausibility for capturing a balance of rigor, depth, and comprehensiveness of the selected topics.

Limitations

The development of a consensus among CEs and medical residents in this study was not without limitations. There was potential for researcher bias stemming from the subjective nature of the process of initial selection of statements and selection of the theoretical frameworks used. Though we used established conceptual models for entrustment decisions and feedback synthesis to structure the construction of our Q sample, the selected theories used to categorically organize our statements may not have been fully representative of all domains of supervision and feedback uptake. Therefore, the statements may not represent an exhaustive list of factors that influence entrustment decisions and feedback receptivity. While this study was limited to emergency CEs and EMRs in Taiwan, we invited a multidisciplinary panel of CEs and medical residents in an attempt to diversify opinions, as suggested by previous literature (38). At the time of writing there is no consensus in the literature defining the optimal panel size and composition when using a Delphi technique with literature, noting panel sizes ranged from 8 to 1,000 of panel members (38). This study invited a panel of 11 CEs and 13 medical residents given the reduced number of statements to maintain homogeneity and reduce oversaturation that could lead to unreliable data. Despite these limitations, the study presents a rigorous approach using strict criteria to establish consensus among CEs and medical residents across various specialties and to construct Q-samples to be used within the context of emergency medicine.

Conclusion

Though not exhaustive, the key factors agreed upon by CEs and medical residents reflect characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. Considering that the factors are consensus-based, they represent a two separate but complementary mindsets between CEs and medical residents. Our study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. As such, our results can serve as a foundation to develop initiatives aimed at the professional development of CEs supervision skills and to enhance medical residents' participation in their own professional development. Additionally, this study aids in increasing transparency when making entrustment-based supervision decisions that can increase feedback receptivity. The incorporation of a Delphi technique further adds to the existing literature and places emphasis on an important tool that can be used in medical education research to rigorously validate Q-statements and develop Q-samples in various specialties.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical approval for this study was granted by the Chang Gung Medical Foundation Institutional Review Board (201900082B0).

Author Contributions

Y-CC: conceptualization, methodology, formal analysis, resources, writing—original draft preparation, writing—review and editing, supervision, project administration, and funding acquisition. RC: methodology, formal analysis, writing—original draft preparation, and writing—review and editing. C-TH: formal analysis, data curation, writing—original draft, writing—review and editing, and project administration. MK: formal analysis, investigation, data curation, and writing—review and editing. NN: conceptualization, methodology, formal analysis, writing—review and editing, and supervision. All authors contributed to the article and approved the submitted version.

Funding

The Ministry of Science and Technology is the Government Ministry of Taiwan for the promotion and funding of Academic Research, Development of Science and Technology and Science Parks (Funding Number: 108-2511-H-182-007-MY2).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors are thankful for all the clinical educators and medical residents who took part in our study. The authors are also grateful for the support provided by Fremen Chihchen Chou, Chung-Hsien Chaou, Hsu-Min Tseng, Chia-Te Kung, and Shu-Chen Liao.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2022.879271/full#supplementary-material

References

1. Ten Cate O. Trust, competence, and the supervisor's role in postgraduate training. BMJ. (2006) 333:748–51. doi: 10.1136/bmj.38938.407569.94

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Hauer KE, Oza SK, Kogan JR, Stankiewicz CA, Stenfors-Hayes T, Cate OT, et al. How clinical supervisors develop trust in their trainees: a qualitative study. Med Educ. (2015) 49:783–95. doi: 10.1111/medu.12745

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Rothwell C, Kehoe A, Farook SF, Illing J. Enablers and barriers to effective clinical supervision in the workplace: a rapid evidence review. BMJ Open. (2021) 11:e052929. doi: 10.1136/bmjopen-2021-052929

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Sterkenburg A, Barach P, Kalkman C, Gielen M, ten Cate O. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med. (2010) 85:1408–17. doi: 10.1097/ACM.0b013e3181eab0ec

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Dijksterhuis MG, Voorhuis M, Teunissen PW, Schuwirth LW, ten Cate OT, Braat DD, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. (2009) 43:1156–65. doi: 10.1111/j.1365-2923.2009.03509.x

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Holzhausen Y, Maaz A, Cianciolo AT, Ten Cate O, Peters H. Applying occupational and organizational psychology theory to entrustment decision-making about trainees in health care: aconceptual model. Perspect Med Educ. (2017) 6:119–26. doi: 10.1007/s40037-017-0336-2

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Govaerts MJ. Trust, entrustment decisions and a few things we shouldn't forget. Perspect Med Educ. (2017) 6:68–70. doi: 10.1007/s40037-017-0337-1

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Mayer RC, Davis JH, Schoorman FD. An integrative model of organizational trust. Acad Manage Rev. (1995) 20:709–34. doi: 10.2307/258792

CrossRef Full Text | Google Scholar

9. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, Ten Cate O. Essential facets of competence that enable trust in graduates: a Delphi study among physician educators in the Netherlands. J Grad Med Educ. (2013) 5:46–53. doi: 10.4300/JGME-D-11-00324.1

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, Ten Cate O. Essential facets of competence that enable trust in medical graduates: a ranking study among physician educators in two countries. Perspect Med Edu. (2013) 2:290–7. doi: 10.1007/s40037-013-0090-z

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Brown SR. Political Subjectivity. Applications of Q Methodology in Political Science. London: Yale University Press (1980).

Google Scholar

12. Babbott S. Commentary: watching closely at a distance: key tensions in supervising resident physicians. Acad Med. (2010) 85:1399–400. doi: 10.1097/ACM.0b013e3181eb4fa4

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Martin P, Copley J, Tyack Z. Twelve tips for effective clinical supervision based on a narrative literature review and expert opinion. Med Teach. (2014) 36:201–7. doi: 10.3109/0142159X.2013.852166

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Pront L, Gillham D, Schuwirth LW. Competencies to enable learning-focused clinical supervision: a thematic analysis of the literature. Med Educ. (2016) 50:485–95. doi: 10.1111/medu.12854

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Johnson C, Molloy E. Building Evaluative Judgement Through the Process of Feedback. Abingdon, NY: Routledge (2018). p. 166–75

Google Scholar

16. Eva KW, Armson H, Holmboe E, Lockyer J, Loney E, Mann K, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract. (2012) 17:15–26. doi: 10.1007/s10459-011-9290-7

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Dochy FJRC. Theories of Learning in the Workplace:Building Blocks for Training and Professional Development Programmes. Abingdon, NY: Routledge (2012).

Google Scholar

18. Duijn C, Welink LS, Mandoki M, Ten Cate OTJ, Kremer WDJ, Bok HGJ. Am I ready for it? students' perceptions of meaningful feedback on entrustable professional activities. Perspect med educ. (2017) 6:256–64. doi: 10.1007/s40037-017-0361-1

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Sandhu G, Thompson-Burdine J, Nikolian VC, Sutzko DC, Prabhu KA, Matusko N, et al. Association of faculty entrustment with resident autonomy in the operating room. JAMA Surg. (2018) 153:518–24. doi: 10.1001/jamasurg.2017.6117

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Tai J, Ajjawi R, Boud D, Dawson P, Panadero E. Developing evaluative judgement: enabling students to make decisions about the quality of work. High Educ. (2018) 76:467–81. doi: 10.1007/s10734-017-0220-3

CrossRef Full Text | Google Scholar

21. Brown B, Gude WT, Blakeman T, van der Veer SN, Ivers N, Francis JJ, et al. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. (2019) 14:40. doi: 10.1186/s13012-019-0883-5

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Watling CJ, Kenyon CF, Schulz V, Goldszmidt MA, Zibrowski E, Lingard L. An exploration of faculty perspectives on the in-training evaluation of residents. Acad Med. (2010) 85:1157–62. doi: 10.1097/ACM.0b013e3181e19722

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Garino A. Ready, willing and able: a model to explain successful use of feedback. Adv Health Sci Educ Theory Pract. (2020) 25:337–61. doi: 10.1007/s10459-019-09924-2

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Ramani S, Könings KD, Ginsburg S, van der Vleuten CP. Feedback redefined: principles and practice. J Gen Intern Med. (2019) 34:744–9. doi: 10.1007/s11606-019-04874-2

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Kornegay JG, Kraut A, Manthey D, Omron R, Caretta-Weyer H, Kuhn G, et al. Feedback in medical education: a critical appraisal. AEM Educ Train. (2017) 1:98–109. doi: 10.1002/aet2.10024

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Iobst WF, Holmboe ES. Programmatic assessment: the secret sauce of effective CBME implementation. J Grad Med Educ. (2020) 12:518–21. doi: 10.4300/JGME-D-20-00702.1

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Cheng W-C, Chen T-Y, Lee M-S. Fill the gap between traditional and new era: the medical educational reform in Taiwan. Ci Ji Yi Xue Za Zhi. (2019) 31:211–6. doi: 10.4103/tcmj.tcmj_229_18

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Chou FC, Hsiao C-T, Yang C-W, Frank JR. “Glocalization” in medical education: a framework underlying implementing CBME in a local context. J Formos Med Assoc. (2021) 13:S0929–6646. doi: 10.1016/j.jfma.2021.10.024

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Fokkema JP, Teunissen PW, Westerman M, van der Lee N, van der Vleuten CP, Scherpbier AJ, et al. Exploration of perceived effects of innovations in postgraduate medical education. Med Educ. (2013) 47:271–81. doi: 10.1111/medu.12081

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Brown SR. A Primer on Q Methodology. Operant Subjectivity. (1993). P. 16.

Google Scholar

31. Watts S, Stenner P. Doing Q methodology: theory, method and interpretation. Qual Res Psychol. (2005) 2:67–91. doi: 10.1191/1478088705qp022oa

CrossRef Full Text | Google Scholar

32. Ward W. Q and You: The Application of Q Methodology in Recreation Research. in Proceedings of the 2009 Northeastern Recreation Research Symposium. New Town Square, PA (2010).

PubMed Abstract | Google Scholar

33. Fontein-Kuipers JACA. Development of a Q-set for a Q-Method study about midwives perspectives of woman-centered care. J.Health Edu. (2016) 1:31–6. doi: 10.15761/HEC.1000107

CrossRef Full Text | Google Scholar

34. Kampen J, Tamás P. Overly ambitious: contributions and current status of Q methodology. Qual Quant. (2013) 48:3109–26. doi: 10.1007/s11135-013-9944-z

CrossRef Full Text | Google Scholar

35. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS ONE. (2011) 6:e20476. doi: 10.1371/journal.pone.0020476

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Green RA. The Delphi technique in educational research. SAGE Open. (2014) 4:1–8. doi: 10.1177/2158244014529773

CrossRef Full Text | Google Scholar

37. John-Matthews JS, Wallace MJ, Robinson L. The Delphi technique in radiography education research. Radiography. (2017) 23:S53–7. doi: 10.1016/j.radi.2017.03.007

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Kirschbaum M, Barnett T, Cross M. Q sample construction: a novel approach incorporating a Delphi technique to explore opinions about codeine dependence. BMC Med Res Methodol. (2019) 19:101. doi: 10.1186/s12874-019-0741-9

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Damio SM editor. Q Methodology, An Overview and Steps to Implementation. Selangor: Asian Journal of University Education (2016).

Google Scholar

40. Van Exel NJA, de Graaf, G,. Q Methodology: A Sneak Preview (2005). Available online at: www.jobvanexel.nl

Google Scholar

41. McKeown B, Thomas D. Q Methodology. Thousand Oaks: Sage Publications, Inc (1988). p. 83.

Google Scholar

42. Paige J, Morin K. Q-Sample construction: a critical step for a Q-Methodological Study. West J Nurs Res. (2014) 38:96–110. doi: 10.1177/0193945914545177

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Zimmerman BJ. Investigating self-regulation and motivation: historical background, methodological developments, and future prospects. Am Educ Res J. (2008) 45:166–83. doi: 10.3102/0002831207312909

CrossRef Full Text | Google Scholar

44. Ali S, Yasmeen R. Practice to preach self-regulation: use of metacognitive strategies by medical teachers in their learning practices. Pak J Med Sci. (2019) 35:1642–6. doi: 10.12669/pjms.35.6.550

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Watts S, Stenner P. Doing Q Methodological Research: Theory, Method Interpretation. London: Sage Publications Ltd. (2012). Available online at: https://methods.sagepub.com/book/doing-q-methodological-research

Google Scholar

46. McKeown M, Hinks M, Stowell-Smith M, Mercer D, Forster J. Q-methodology, risk training and quality management. Int J Health Care Qual Assur. (1999) 12:254–66. doi: 10.1108/09526869910291823

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Williams AL, Blomkalns AL, Gibler WB. Residency training in emergency medicine: the challenges of the 21st century. Keio J Med. (2004) 53:203–9. doi: 10.2302/kjm.53.203

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Ryan JG, Mandel FS, Sama A, Ward ME. Reliability of faculty clinical evaluations of non-emergency medicine residents during emergency department rotations. Acad Emerg Med. (1996) 3:1124–30. doi: 10.1111/j.1553-2712.1996.tb03372.x

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. (1995) 311:376–80. doi: 10.1136/bmj.311.7001.376

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Raskin MS. The delphi study in field instruction revisited: expert consensus on issues and research priorities. J Soc Work Educ. (1994) 30:75–89. doi: 10.1080/10437797.1994.10672215

CrossRef Full Text | Google Scholar

51. Rayens MK, Hahn EJ. Building consensus using the policy delphi method. Policy Polit Nurs Pract. (2000) 1:308–15. doi: 10.1177/152715440000100409

CrossRef Full Text | Google Scholar

52. von der Gracht HA. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change. (2012) 79:1525–36. doi: 10.1016/j.techfore.2012.04.013

CrossRef Full Text | Google Scholar

53. Brown DR, Warren JB, Hyderi A, Drusin RE, Moeller J, Rosenfeld M, et al. Finding a path to entrustment in undergraduate medical education: a progress report from the AAMC Core entrustable professional activities for entering residency entrustment concept group. Acad Med. (2017) 92:774–9. doi: 10.1097/ACM.0000000000001544

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Ten Cate O, Hart D, Ankel F, Busari J, Englander R, Glasgow N, et al. Entrustment decision making in clinical training. Acad Med. (2016) 91:191–8. doi: 10.1097/ACM.0000000000001044

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Kovatch KJ, Prince MEP, Sandhu G. Weighing entrustment decisions with patient care during residency training. Otolaryngol Head Neck Surg. (2018) 158:1024–7. doi: 10.1177/0194599818764652

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Prost E. toward a shared mental model of progressive competence in postgraduate medical education. Can Med Educ J. (2018) 9:e115–8. doi: 10.36834/cmej.43389

CrossRef Full Text | Google Scholar

57. Schwartz A, Balmer DF, Borman-Shoap E, Chin A, Henry D, Herman BE, et al. Shared mental models among clinical competency committees in the context of time-variable, competency-based advancement to residency. Acad Med. (2020) 95:S95–102. doi: 10.1097/ACM.0000000000003638

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Fonteijn H editor. Making Students Responsible for Their Learning@ Empowering Learners to Build Shared Mental Models. Springer International Publishing (2015).

Google Scholar

59. Weller JM, Naik VN, San Diego RJ. Systematic review and narrative synthesis of competency-based medical education in anaesthesia. Br J Anaesth. (2020) 124:748–60. doi: 10.1016/j.bja.2019.10.025

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Gercama A, van Lankveld T, Kleinveld J, Croiset G, Kusurkar R. Bridging the gap between CBME in theory and practice: the role of a teacher community. Perspect Med Educ. (2014) 3:486–91. doi: 10.1007/s40037-014-0147-7

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Mulder H. ter Braak E, Chen HC, ten Cate O. Addressing the hidden curriculum in the clinical workplace: a practical tool for trainees and faculty. Med Teac. (2019) 41:36–43. doi: 10.1080/0142159X.2018.1436760

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Holmboe ES, Ward DS, Reznick RK, Katsufrakis PJ, Leslie KM, Patel VL, et al. Faculty development in assessment: the missing link in competency-based medical education. Acad Med. (2011) 86:460–7. doi: 10.1097/ACM.0b013e31820cb2a7

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Wood TJ. Exploring the role of first impressions in rater-based assessments. Adv Health Sci Educ Theory Pract. (2014) 19:409–27. doi: 10.1007/s10459-013-9453-9

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Suhoyo Y, Hell E, Kerdijk W, Emilia O, Schönrock-Adema J, Kuks J, et al. Influence of feedback characteristics on perceived learning value of feedback in clerkships: does culture matter? BMC Med Educ. (2017) 17:1–7. doi: 10.1186/s12909-017-0904-5

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract. (2008) 13:275–88. doi: 10.1007/s10459-006-9039-x

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Kilminster S, Cottrell D, Grant J, Jolly B. AMEE. Guide No. 27: effective educational and clinical supervision. Med Teach. (2007) 29:2–19. doi: 10.1080/01421590701210907

PubMed Abstract | CrossRef Full Text | Google Scholar