INteractive Virtual Expert-Led Skills Training: A Multi-Modal Curriculum for Medical Trainees

Background: Internationally, pediatric depression and suicide are significant issues. Additionally, in the context of the COVID-19 pandemic, pediatric mental health needs are rising astronomically. In light of Child & Adolescent Psychiatrist (CAP) subspecialist shortages in the United States (US), there is an increasing call for primary care physicians in Family Medicine and Pediatrics to address an increasingly broad variety of patient needs. Here we report on the development and preliminary evaluation of medical student and resident perceptions on the “INteractive Virtual Expert-led Skills Training” (INVEST) medical education curriculum, a virtual synchronous CAP curriculum employing active learning strategies, including expert-led discussion and video modeling, and discussion designed to meet those priorities.

Methods: In a standardized 60-min training format, our curriculum leverages audience response system polling, video modeling of key clinical skills, and interactive discussion with an expert subspecialist, over a virtual video conferencing platform. The primary educational strategy relies on use of video modeling to demonstrate best practice with CAP led group discussion to solidify and explain important concepts. Five waves of medical students and residents (N = 149) participated in the INVEST curriculum and completed pre- and post-training surveys regarding knowledge and comfort in the management of pediatric patients with depression and suicidality.

Results: Trainee participants reported significant positive gains in perceived likelihood of encountering pediatric suicidality as well as knowledge/comfort with depression screening and suicidality assessment in a primary care setting. Across some competency areas, there was an effect of medical learner level. Learners at lower levels generally reported the highest benefit. Medical students reported significant increases in their comfort interpreting and discussing positive depression screens and evidenced the greatest relative benefit in comfort with discussing suicidality.

Conclusion: To our knowledge, INVEST is the first fully virtual, multimodal curriculum led by expert CAP subspecialists. Our findings suggest that INVEST shows promise for equipping medical learners with baseline knowledge for caring for patients with pediatric depression and suicidality. This synchronous, virtually delivered curriculum allows for critical training delivered to diverse medical learners regardless of geographic location, a particular benefit during the current COVID-19 pandemic.

Introduction

Internationally, pediatric depression and suicide are significant issues. Depression impacts ~6.2% of children and adolescents ages 5–17 years (1), resulting in impact to school performance and relationships. Compared to adults, diagnosis and treatment for children and adolescents are much more challenging (2) and subsequently only half of adolescents with depression are diagnosed before adulthood (3). While for individuals 15–29 years of age it is well documented that suicide is the second leading cause of death (4), information on suicide in pre-adolescents is less readily known; one study estimated the mean rate of suicide in children and young adolescents up to 14 years is ~0.6/100.00 worldwide (5). Additionally, in the context of the COVID-19 pandemic, pediatric mental health needs are rising astronomically (6). With the staggering impact of depression and suicidality on children and teens, identification and treatment are critical. The need now, more than ever, for virtual trainings (7) which target these skill areas, allowing access to real-world situations in low risk environments (8, 9) across diverse geographic locations (10) has become urgent.

However, severe workforce shortages exist in the Child & Adolescent Psychiatry (CAP) workforce (11, 12), which impact patient access to quality care (13–16) particularly in rural areas (11, 15). In light of CAP subspecialist shortages in the United States (US), there is an increasing call for primary care physicians in Family Medicine and Pediatrics to address an increasingly broad variety of patient needs, including mental health conditions such as depression and suicidality (14, 17, 18). However, primary care providers (18, 19) and residents in primary care residency training programs (17, 20, 21) report inadequate knowledge and skill in the assessment, diagnosis and treatment of these areas. To care for these patient needs, medical providers must acquire basic competence in pediatric mental and behavioral health (14, 22), including development of both interpersonal communication skills (23) and content knowledge (24–26).

Targeted curricula that employ novel training modalities (27–30) that make use of the available tools (31–33) are needed to make optimal use of the limited CAP workforce available as teaching experts (12, 13, 15, 16, 34). The “Guidelines for Adolescent Depression in Primary Care” for instance address steps to practice transformation to initiate identification and initial management of adolescents with depression in the primary care setting (31). In the United States, the American Academy of Child & Adolescent Psychiatry has proposed “Best Principles for Integration of Child Psychiatry into the Pediatric Health Home” focused on principles to apply the medical home model to psychiatric care needs for this population (35). Other tools, such as “Project TEACH's free library” of mental health screen “Resources”¹, the American Academy of Pediatrics “Mental Health Toolkit” ², and the PHQ-9 depression screen, are additionally being shared virtually to support practice expansion to care for the mental health needs of children and adolescents. While all of these guidelines and tools offer practices the ability to self-guide with evidence-based information around pediatric depression and suicidality, there is still a role for interactive training to build competency for providers.

To address the need for new ways of supporting primary care skill development to address the soaring pediatric depression and suicidality rates (1, 4, 5) in light of the international CAP workforce limitations (11, 12, 36), our team sought to develop a virtually-delivered medical education curriculum on the care of these patients. This virtual training was meant to suit a traditional 60-min didactic teaching block for medical students and residents in the primary care field of Family Medicine. The team's goal was to improve the awareness and comfort in responding to positive screening results of pediatric depression and suicidality in the outpatient setting of participants. Here we report on the development and preliminary evaluation of medical student and resident perceptions on the “INteractive Virtual Expert-led Skills Training” (INVEST) medical education curriculum (here after referred to as our curriculum), a virtual synchronous CAP curriculum employing active learning strategies (20, 37, 38) of expert-led video modeling, and discussion designed to meet those priorities (39).

Methods

This study was determined to be “Protocol Exempt” following review by Indiana University IRB (approved 2/17/2020; reference number 2001953926A001). All methods were performed in accordance with the relevant guidelines and regulations. Information on the voluntary nature of the research study for informed consent was provided on all pre- and post-surveys.

Development of INVEST Curriculum

Developed by a multidisciplinary team of subspecialists at Indiana University School of Medicine (IUSM), our curriculum was designed as a fully virtual multimodal training curriculum (8, 17, 19, 22, 24, 30, 40, 41) delivered by CAP faculty members. Multimodal trainings (20, 22, 42) use a variety of curriculum delivery methods in a single training (e.g. video modeling, readings, didactic sessions, expert feedback, etc.) and have demonstrated longer lasting impact on skill development compared to traditional, lecture based education (42–45) with improved patient outcomes (44, 46), enhanced clinician skills in discussion of sensitive topics (9, 47–49), enhanced engagement with vulnerable patients (20, 50, 51), and greater facilitation of linkages with community support resources (9). Our curriculum format was grounded in adult learning theory to address the varied experiences in the clinic and classroom of medical students and residents, building new information over their existing scaffold, and supporting lifelong learning tendencies (19, 52, 53). Ongoing financial and labor investment (38) were minimized through the development of a single structure that could be given by multiple personnel in the field without significant curriculum specific training. The curriculum structure is designed to promote active learning in primary care students and residents through particular focus on the (1) impact of mental and behavioral health condition(s), (2) available screening tool(s), and (3) patient interviewing and feedback skills for use in a primary care outpatient clinical setting. Our curriculum provides educational content (20, 50) on (1) pediatric depression, including prevalence, symptoms, and key differences between pediatric and adult presentation, and (2) suicidal behavior, including features of suicidal ideation, differences between ideation and attempt, risk factors, and strategies for discussing suicidality with patients (Supplementary Material).

In a standardized 60-min training format, our curriculum leverages audience response system polling, video modeling of key clinical skills, and interactive discussion with an expert subspecialist, over a virtual video conferencing platform (see Figure 1). The primary educational strategy relies on use of video modeling to demonstrate best practice with CAP led group discussion to solidify and explain important concepts. Our team produced two recorded video models of outpatient pediatric primary care appointments with a patient actor that demonstrated behavioral health assessment competencies as well as coordination and communication between the physician, patient, and office support staff. The first 10-min video models a clinician interviewing an adolescent patient with a positive Patient Health Questionnaire-9 (PHQ-9)³, which is widely-used in primary care for depression screening, augmented by discussion on the PHQ-9 results and interpretation, key history taking for depression diagnosis, and discussion of symptoms unique to pediatric patients. The second 7-min video models a clinician addressing a positive PHQ-9 with endorsement of suicidality symptoms, including performing a safety assessment and consideration of how to discuss need for disclosure to the adolescent's caregivers. These video modeling examples lay the groundwork for interactive discussion between learners and the CAP subspecialist, used in conjunction with virtual polling to address common misperceptions around pediatric depression and suicidality, and a limited number of content slides to ensure all attendees received an overview of pediatric depression and suicidality.

FIGURE 1

Figure 1. INVEST Curriculum. Content above delivered by faculty boarded in Child & Adolescent Psychiatry. Supplemental work was performed by the co-facilitator, including chat box technical assistance, trainee engagement, and collecting trainee questions and concerns.

Our curriculum was initially delivered to 30 medical students and residents; pre- and post-training surveys were collected (see survey description below) and qualitatively used to improve the curriculum. Adaptations made following this trial included: (1) addition of a co-facilitator to provide real-time technical assistance, promote discussion via virtual platform chat box, and facilitate and manage discussion flow, (2) creation of discussion prompts (tailored for student or resident learners), (3) use of a mid-point discussion break in the first 10 min video to promote additional discussion, and (4) implementation of a post-training follow-up email with key curriculum points and resources (see Figure 1 for a flow chart of INVEST).

The final curriculum included a primary presenter and a co-facilitator who provided education as a team. While the topics of pediatric depression and suicide were consistently taught via videos, polling questions, content slides, and discussion questions (set at level for either residents or medical students), each time the training was given some variability occurred during discussion based on questions and/or comments from the participants. Secondary to a total 60-min available for the training, each of these components was time limited. The primary presenter focused on delivery of content including running the polling software, facilitating group discussion around the video models, and delivering content (e.g., key points). The co-facilitator provided technological support throughout the presentation by helping participants who logged on after the presentation had started, collecting chat box questions (public and private) which the primary presenter could not see in presentation mode, supporting group engagement via chat box prompts, and anticipating or responding to technical issues. A follow up email with a content summary and clinical resources was sent to all attendees following the close of the presentation.

Assessment of INVEST Curriculum

Following Cook and colleagues' (39) framework for medical education research, a modified justification study design was used to assess learner perceptions about curriculum impact. Although justification designs often compare one educational intervention to another, a first step in assessing intervention efficacy is often to examine pre-post changes across a number of different learning domains (e.g., learner knowledge/attitude, clinical diagnosis, clinical interventions, teaching and leadership, assessment). In the present study, we focus on evaluating learner changes in knowledge and attitudes following engagement in the training. In order to measure this outcome, a self-report survey was developed to assess change across key learning objectives of the our curriculum. The survey consisted of seven items scored on a five-point Likert scale (see Table 2 for survey items). Likert scale anchors for survey items are as follows: Items 1/2: 1 (Very unlikely/rarely) - 5 (Very likely/at least weekly); Items 3/6: 1 (Nothing) - 5 (A great deal/I could explain it to others); Items 4/5/7: 1 (Very uncomfortable/I'd try to avoid doing it) - 5 (Very comfortable/would not hesitate).

Participants

Our curriculum was delivered to medical students and Family Medicine (FM) residents across two institutions in five waves from 5/1/2020 to 6/30/2020. In all trainings, the presenters were generally unknown to participants. Curriculum participants were recruited through (1) medical student special interest group events (Pediatrics, Neurology, Psychiatry) which were open to any student, and (2) FM program directors seeking opportunities for resident didactic education in pediatrics and mental and behavioral health. In total, 170 participants completed our curriculum; 149 participants completed pre- and post-surveys (response rate = 88%). See Table 1 for participant characteristics.

TABLE 1

Table 1. Participant characteristics: graduate degree, medical training level, and medical specialty area of the participants (N = 149).

Data Collection and Analysis

Pre- and post-training surveys were collected electronically via REDcap (54, 55). Non-parametric statistical analyses were completed using SPSS Statistics 27.0 (56). We explored differences from pre- to post-training for all participants using Wilcoxan Signed Rank tests. To examine the effect of medical trainee level, participants were grouped based on clinical skill development: MS I-II (n = 30); MS III (n = 29); MS IV (n = 49); PGY I (n = 19); PGY II-III (n = 21). To explore effects of medical student intended specialty, two groups were created: (1) primary care (n = 36), and (2) specialty care (n = 15); see Table 1 for description of specialty grouping [Insert Table 1]. Using Kruskal-Wallis tests, we examined the effects of medical trainee level and intended medical specialty on change in scores from pre- to post-training by creating difference scores for each survey item. When a significant main effect was present, we conducted post hoc analyses using Mann-Whitney tests and Wilcoxan Signed Rank tests.

Results

See Table 2 for item-level descriptive statistics for both pre- and post-scores for all participants, and participants by medical trainee level (student or resident year) and medical specialty. See Table 3 for item-level descriptive statistics for difference scores by training level and medical specialty.

TABLE 2

Table 2. Item-level survey descriptive statistics for all participants (N = 149) by level and medical specialty area.

TABLE 3

Table 3. Item-level descriptive statistics for difference scores by participant level and intended medical specialty area (N = 149).

Item 1: How Likely Do You Think You Are to Encounter Child/Adolescent Patients With Depression?

There was no significant difference pre- to post- training in the perceived likelihood of encountering children/adolescents with depression (Z = 815.00, p = 0.07). There was a significant effect of medical trainee level on difference scores [H (4) = 17.74, p = 0.001]. Pairwise comparisons revealed that MS I-II showed significantly greater difference scores than all other training levels (all p < 0.046); no other pairwise comparisons across training levels were significant (all p > 0.16). A significant difference was found from pre- to post-training for MS I-II (Z = 114.00; p = 0.001) such that MS I-II participants rated perceived frequency higher following the training. This finding was not significant for any other training level (all p > 0.11).

Item 2: How Likely Do You Think You Are to Encounter Child/Adolescent Patients With Suicidal Thoughts/Feelings/Behaviors?

There was a significant difference pre- to post- training in participants' perceived likelihood of encountering children/adolescents with suicidal thoughts/feelings/behaviors (Z = 1652.00, p = < 0.001) indicating that participants rated perceived frequency higher following delivery of our curriculum. There was no significant effect of training level [H (4) = 2.07, p = 0.72] on difference scores.

Item 3: How Much Do You Know About Depression in Children/Adolescents?

There was a significant difference pre- to post- training in perceived knowledge about depression in children/adolescents (Z = 1,652.00, p = < 0.001) with participants rating perceived knowledge higher following the training. There was no significant effect of medical trainee level [H (4) = 2.16, p = 0.71] on difference scores.

Item 4: How comfortable Do You Feel Interpreting a Depression Screener?

There was a significant difference pre- to post- training in participants' comfort interpreting a depression screener (Z = 4,379.50, p = < 0.001) with comfort increasing following participation in our curriculum. There was a significant effect of medical trainee level on difference scores [H (4) = 12.61, p = 0.01]. Pairwise comparisons revealed significant variation in difference scores between MS I-III and PGY I-III as well as between MS IV and PGY II-III (all p < 0.04) with lower medical trainee levels showing greater gains in comfort than higher training levels. No other pairwise comparisons were significant (all p > 0.10). A significant difference was found from pre- to post- training for MS I-II (Z = 238.00; p = 0.002), MS III (Z = 231.00; p < 0.001), and MS IV (Z = 572.00; p < 0.001), such that participants across these medical trainee levels rated their comfort higher following participation in our curriculum. This finding was not significant for PGYI-III (all p >0.08).

Item 5: How Comfortable Do You Feel Discussing a Positive Screen?

There was a significant difference pre- to post- training in participants' comfort discussing a positive depression screener with patients (Z = 2,854.00, p = < 0.001) with comfort increasing following the training. Additionally, there was a significant effect of medical trainee level on difference scores [H (4) = 16.00, p = 0.003). Pairwise comparisons revealed significant variation in difference scores between MS I-IV and PGY II-III as well as between MS I-II and PGY I (all p < 0.03) with lower medical trainee levels showing greater gains in comfort than higher medical trainee levels. No other pairwise comparisons were significant (all p > 0.10). A significant difference was found from pre- to post- training for MS I-II (Z = 204.00; p < 0.001), MS III (Z = 121.00; p = 0.003), and MS IV (Z = 314.50; p < 0.001), with these participants rating their comfort higher following participation in our curriculum. This finding was not significant for PGY I-III (all p > 0.25).

Item 6: How Much Do You Know About Performing a Suicidality Assessment?

There was a significant difference pre- to post- training in participants' perceived knowledge about performing a suicidality assessment (Z = 6,186.00, p = < 0.001) with participants reporting increased knowledge following participation in our curriculum. There was no significant effect of medical trainee level [H (4) = 1.22, p = 0.88) [H (1) = 0.003, p = 0.96) on difference scores.

Item 7: How Comfortable Do You Feel Discussing Thoughts/Feelings Around Suicidal Thoughts/Feelings/Behaviors in Children/Adolescents?

There was a significant difference pre- to post- training in participants' comfort discussing suicidal thoughts/feelings/behaviors in children/adolescents (Z = 3,940.00, p = < 0.001) with comfort increasing following participation in our curriculum. A significant effect of medical trainee level on difference scores [H (4) = 9.57, p = 0.048) was found. Pairwise comparisons revealed that MS I-II showed greater gains in comfort as compared to all other levels (all p < 0.05); no other pairwise comparisons were significant (all p > 0.33). However, a significant difference was found from pre- to post- training for MS I-II (Z = 268.50; p < 0.001), MS III (Z = 136.00; p = 0.002), MS IV (Z = 394.00; p < 0.001), PGY I (Z = 66.00; p = 0.02), and PGY I-III (Z = 56.00; p = 0.03), indicating that while MS-I-II showed greater gains, all participants rated comfort higher following participation in our curriculum.

All Items

There was no significant effect of specialty area on difference scores (all p > 0.06).

Discussion

With an overall paucity internationally of CAP subspecialists (13–16, 34, 36) and the critical rates of pediatric depression and suicidality (1, 3, 4), there is a critical need for primary care clinicians to develop competency in screening and basic assessment of conditions traditionally managed by subspecialists, including mental and behavioral health disorders such as depression and suicidality (8, 14, 18, 19, 48, 57). To address this need, we developed our INVEST curriculum, a virtually-delivered, synchronous medical education program (8, 17, 19, 22, 24, 30, 40, 41) leveraging mixed technology (10) to bring subspecialty content to a diverse audience of medical students and residents. We applied our multimodal structure to focus on delivery of key skills in assessing pediatric depression and suicidality in the primary care setting (20, 50) through video modeled best practice with CAP subspecialist-led discussion. To our knowledge, this is the first curriculum that integrates multi-modal, active learning strategies (i.e., video modeling, interactive case discussion, virtual polling, subspecialty facilitation) in a fully virtual platform to deliver active education in CAP in a standard 60-min format. In creating our curriculum structure, concrete steps are demonstrated, discussed, and reviewed to address positive PHQ-9 screening results within an outpatient setting, serving as a guide for primary care learners in medical school and residency.

Outcome data collected across five waves of implementation suggests that, overall, medical students and residents experienced significant improvements in perceived knowledge and comfort with provision of care around pediatric depression and suicidality. As a result of the training, medical students reported significant improvement in awareness of pediatric depression as a clinical issue that is likely to be encountered regularly in the primary care setting. We found significant positive gains from pre- to post- training in medical students' and residents' perceived likelihood of encountering pediatric patients with suicidal ideation, knowledge about pediatric depression and suicidality, and comfort in administering and responding to depression screenings and suicidality assessment. While we failed to find differences in benefit of our curriculum across medical student intended specialty area, we did find an effect of medical learner level across a number of competency areas. Medical students and residents in lower educational levels generally reported the highest benefit. Specifically, medical students reported significant increases in their comfort interpreting and discussing positive depression screens and evidenced the greatest relative benefit in comfort with discussing suicidality. Together, these findings suggest that our curriculum may be particularly effective in preparing learners to enter clinical experiences where they may encounter patients with these behavioral health challenges (e.g., medical students before entering the outpatient setting, residents during orientation to training). While medical students reported greater gains in knowledge/comfort addressing suicidal thoughts/feelings/behaviors in pediatric patients, residents also endorsed perceived benefit. Interestingly, residents did endorse benefit from training in pediatric suicidality (items 2, 6, 7), suggesting that this may be a particular area of weakness for medical students and residents alike, consistent with findings from Fallucco et al. (17).

There are several possible reasons that more experienced learners (i.e., PGY2 or PGY3 residents) reported fewer gains from our curriculum, such as prior exposure to adult content or techniques (e.g., giving bad news) (29, 47, 49) in their clinical rotations, making depression content more familiar to them. It should be noted that in group discussion, resident level participants generally denied previous personal experience with pediatric patients showing symptoms of depression or suicidality. Additionally, evidence suggests that primary care graduates may perceive diagnosis and treatment of mental health issues as the responsibility of the subspecialist (19); as a result, they may have perceived the content as less relevant, impacting their own awareness of skills and attitudes in this area. It is possible as well that since the trainers were unknown to the residents and residents were required to attend the educational activity that there was more difficulty in building rapport in the virtual environment. Additionally, it should be noted that many residents who participated joined from their clinical service areas (e.g., hospital work room space) and may have had divided attention because of their caretaking responsibilities during the activity. It should be noted that in the cohort of medical students, a significant number of individuals were recruited via special interest groups. Therefore, the medical student audience may have been more willing to engage in the virtual training modality, as those individuals likely were already interested in the topics or population. These individuals also may have been more willing to report improvements in their understanding of the topics covered because of this interest.

Initial outcome data is promising, however there are several limitations to our approach that warrant discussion. First, outcomes were targeted to the learning objectives such that other needs/gains may have been missed. Further, outcomes were measured via participant self-report only, prohibiting study of objective change in clinical competence, and only pre- and immediate post-data was collected (total 2 data points). Collection of data at a third time point (i.e., follow-up) would strengthen the study and allow for generalization regarding maintenance of gains over time. Collection of qualitative information from participants as well as measures of engagement or related constructs from facilitators would have allowed for richer outcome data. While we did collect and analyze the effects of medical trainee level and intended specialty area, we did not collect demographic information on to minimize the burden of questionnaires for participants; this will be important in the future. Finally, further adaptation of our curriculum to teach pediatric depression and suicidality information is warranted. As all groups reported increased knowledge regarding suicidality topics, it may be beneficial to focus on this topic through development of a stand-alone module, emphasize the relationship between depression and suicidality, and/or share common content misconceptions.

Benefits and Challenges of Virtual Delivery of Medical Education

Our curriculum provided learners with a succinct introduction to a complex topic with multi-modal and activity learning strategies. Participants additionally had the opportunity to have direct contact with CAP subspecialists, who would otherwise have been inaccessible during their medical trainee years (11, 12). While CAP subspecialist time was still required for the 60 min trainings, the overall time required was minimized by eliminating travel (+2 h to some sites) and development of a multimodal (20, 22, 42) structure (i.e., videos, polling), which allowed the CAP educator to customize conversations and discussion to the needs of the attendees while optimizing individual efforts. Using CAP subspecialists for medical training in this way may increase primary care workforce competence in subspecialty areas with low access/high need. CAP subspecialists are particularly well-equipped to address very challenging cases brought by more senior learners (PGY2 or PGY-3) to discussion, build pediatric content on the foundation.

An additional strength of our curriculum is the completely virtual delivery paired with synchronous teaching and direct engagement with CAP subspecialists (22, 37, 40). This educational format provides the opportunity for active (20, 37, 38) learning while individuals remain socially distanced, a necessity during the time of COVID-19. The virtual format also expands educational reach to diverse geographic locations, across programs or institutions, as well as across education levels or specialty areas. Though our curriculum was delivered to medical students and residents, the format can be applied to continuing medical education around subspecialty topics, including new or updated best practices for established providers.

Despite clear benefits to developing and deploying virtual medical education curricula, unique challenges exist for educators. For instance, participants may tune in from disruptive environments (e.g., while driving or from shared clinical workspace). We found that medical students and residents who shared a log-in or who joined without video were less likely to actively participate with the group and did not add comments or questions verbally or via chat box. Engagement was enhanced when participants were provided with instructions about the virtual platform prior to the training, including the directive to log on individually with camera and microphone. Additionally, the co-facilitator significantly contributed to engagement by sharing technical information, initiating, and elaborating on discussion topics, and managing participation via the virtual platform chat box.

Future Directions

In the future, it will be important to measure changes in clinical competency following engagement in our curriculum. Other training programs have used post-training online cases with virtual patients (58) or standardized patient-based simulation (48, 59) as a means of observing clinician skill development. Chart review of referral practice, screening tool use or other skills could be another means of looking at practice changes following training and could be made into a senior level quality improvement project for practices or residents. Further, as discussed by Cook, Bordage, & Schmidt, investigation is also needed around how the multimodal format of our curriculum directly compares to other training methods addressing pediatric depression and suicidality (39). In particular, traditional didactic and more active formats like patient-based simulation should be compared, as well as objective examination of the mechanisms of learning change following engagement in the curriculum.

Future adaptations of our curriculum should target other high-need areas for primary care, including other content areas such as Autism Spectrum Disorder or Intellectual Disability or to enhance psychosocial skill training across diagnoses in the clinical environment [e.g., see Roter et al. (23) for further description of how teaching communication and psychosocial skills to address patient psychosocial distress can improve resident diagnostic accuracy and management]. Finally, our curriculum could be expanded to address practice quality improvement issues (e.g., implementing screening tools into practice) while addressing concerns about time and reimbursement [e.g., see Horwitz et al. (18)].

To improve our model, additional methods of content building, particularly for senior level residents (PGY-2 or PGY-3), and engagement must be considered. Senior level residents may benefit from additional responsibility during training, such as by bringing clinical examples, relating material to specific clinic encounters, directing learning with prepared questions, or taking the place of the co-facilitator to support discussion and technical assistance. Our curriculum could also be integrated with other forms of skill application for senior level residents, such as through facilitated role play, recorded encounters with subsequent self-rated practice (17, 20, 41), pediatric simulation (60), or unannounced standardized patients (20, 21) for formative or summative assessment (7, 28). Further, residents may be better engaged with other adaptations, such as inclusion of a familiar faculty member or a more senior resident, in order to more effectively support trainees in their exploration of knowledge gaps or feelings of vulnerability regarding difficult cases. Integrating additional tools such as skills checklists for observers to complete while viewing the video models (9, 17, 21, 27, 41) may also enhance engagement.

Conclusions

Given the international dearth of CAP subspecialists, medical professionals will ultimately be faced with assessing and treating conditions traditionally managed by these clinicians. Thus, it is critical to develop and test effective, efficient, and financially feasible methods of training in these areas. To our knowledge, our curriculum “INVEST” is the first fully virtual, synchronous multimodal curriculum led by expert subspecialists in the CAP field which adheres to a traditional 60-min format. Our findings suggest that our curriculum shows promise for equipping medical students and residents with baseline knowledge when piloted to teach pediatric depression and suicidality. While further adaptation and evaluation is necessary, it is our hope that our curriculum framework can serve as a model for a broad array of virtual medical education content areas, with the ultimate goal of building medical students' and residents' interest and competence in primary care as well as underserved pediatric subspecialties.

¹https://projectteachny.org/resources/

²https://toolkits.solutions.aap.org/mental-health/home

³https://www.phqscreeners.com/images/sites/g/files/g10060481/f/201412/PHQ-9_English.pdf

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by Indiana University IRB reference number 2001953926A001. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

MC, JD, and AH developed and provided all trainings. MC drafted the initial manuscript and took primary responsibility for the manuscript development. EC managed all REDcap development and management and contributed to manuscript editing. BE and RM conducted data analysis and interpretation. AH, BE, and RM provided critical review and intellectual contributions to all manuscript drafts. All authors contributed to the article and approved the submitted version.

Funding

Clinician salary support for this research came through the Riley Childrens Foundation and Kiwanis Indiana Three Wishes Campaign for MC. RM time was supported by the Robert and Helen Haddad Family Foundation Scholarship and the Riley Children's Foundation.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Our team is grateful to the mentorship of Dr. Mary Ciccarelli, director of the Kiwanis Indiana Three Wishes Campaign team, and for the collaboration of teaching programs in the state of Indiana who allowed our team to bring this model to their trainees: IU Arnett Family Medicine Residency, Indiana University School of Medicine Family Medicine Residency at Memorial Hospital And Health Care Center, IU Health Ball Family Medicine Residency, IU Methodist Family Medicine Residency, and Marian University College of Osteopathic Medicine. We would like to also thank Indiana University's Instructional Technology team and Simulation Center for making these video models possible.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2021.671442/full#supplementary-material

References

1. Erskine HE, Baxter AJ, Patton G, Moffitt TE, Patel V, Whiteford HA, et al. The global coverage of prevalence data for mental disorders in children and adolescents. Epidemiol Psychiatr Sci. (2017) 26:395–402. doi: 10.1017/S2045796015001158

PubMed Abstract | CrossRef Full Text | Google Scholar

2. US Food & Drug Administration. Don't Leave Childhood Depression Untreated. Available online at: https://www.fda.gov/consumers/consumer-updates/fda-dont-leave-childhood-depression-untreated (accessed May 12, 2021).

3. Kessler RC, Avenevoli S, Ries Merikangas K. Mood disorders in children and adolescents: an epidemiologic perspective. Biol Psychiatry. (2001) 49:1002–14. doi: 10.1016/S0006-3223(01)01129-5

PubMed Abstract | CrossRef Full Text | Google Scholar

4. World Health Organization. Suicide. Available online at: https://www.who.int/news-room/fact-sheets/detail/suicide (accessed May 5, 2021).

5. Dervic K, Brent DA, Oquendo MA. Completed suicide in childhood. Psychiatr Clin North Am. (2008) 31:271–91. doi: 10.1016/j.psc.2008.01.006

CrossRef Full Text | Google Scholar

6. Fegert JM, Vitiello B, Plener PL, Clemens V. Challenges and burden of the Coronavirus 2019 (COVID-19) pandemic for child and adolescent mental health: a narrative review to highlight clinical and research needs in the acute phase and the long return to normality. Child Adolesc Psychiatry Ment Health. (2020) 14. doi: 10.1186/s13034-020-00329-3

PubMed Abstract | CrossRef Full Text

7. Kaltman S, Talisman N, Pennestri S, Syverson E, Arthur P, Vovides Y. Using technology to enhance teaching of patient-centered interviewing for early medical students. Simul Healthc. (2018) 13:188–94. doi: 10.1097/SIH.0000000000000304

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Cole S, Raju M, Barrett J, Gerrity M, Dietrich A. The macarthur foundation depression education program for primary care physicians: background and rational. Gen Hosp Psychiatry. (2000) 22:299–358. doi: 10.1016/S0163-8343(00)80007-9

CrossRef Full Text | Google Scholar

9. Niebruegge B, Holbrook JM, Vernon C, Grotton C, Maric A. The future of population medicine: investigating the role of advanced practice providers and simulation education in special patient populations. Dis Mon. (2019) 65:24. doi: 10.1016/j.disamonth.2018.09.005

PubMed Abstract | CrossRef Full Text

10. Moran J, Briscoe G, Peglow S. Current technology in advancing medical education: perspectives for learning and providing care. Acad Psychiatry. (2018) 42:796–9. doi: 10.1007/s40596-018-0946-y

PubMed Abstract | CrossRef Full Text | Google Scholar

11. American Academy of Child and Adolescent Psychiatry. Workforce Maps by State. Available online at: https://www.aacap.org/aacap/Advocacy/Federal_and_State_Initiatives/Workforce_Maps/Home.aspx (accessed July 21, 2020).

12. Freed GL, Moran LM, Van KD, Leslie LK. Current workforce of pediatric subspecialists in the United States. Pediatrics. (2017) 139:e20163604. doi: 10.1542/peds.2016-3604

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Keller DM, Davis MM, Freed GL. Access to pediatric subspecialty care for children and youth: possible shortages and potential solutions. Pediatr Res. (2020) 87:1151–2. doi: 10.1038/s41390-020-0889-3

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Skokauskas N, Fung D, Flaherty LT, von Klitzing K, Puras D, Servili C, et al. Shaping the future of child and adolescent psychiatry. Child Adolesc Psychiatry Ment Health. (2019) 13:1–7. doi: 10.1186/s13034-019-0279-y

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Shugerman R, Shipman S, Cull W, Rimsza M, O'Connor K, Pletcher B. Too Many, Too Few, or Just Right? A National Survey of Practicing Pediatricians Assessing the Pediatric Subspecialty Supply. Honolulu: Pediatric Academic Societies Annual Meeting (2008).

16. Steinman KJ, Shoben AB, Dembe AE, Kelleher KJ. How long do adolescents wait for psychiatry appointments? Community Ment Health J. (2015) 51:782–9. doi: 10.1007/s10597-015-9897-x

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Fallucco EM, Hanson MD, Glowinski AL. Teaching pediatric residents to assess adolescent suicide risk with a standardized patient module. Pediatrics. (2010) 125:9. doi: 10.1542/peds.2009-2135

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Horwitz SM, Storfer-Isser A, D KB, Szilagyi M, Garner A, O'Connor KG, et al. Barriers to the identification and management of psychosocial problems: Changes from 2004 to 2013. Acad Pediatr. (2015) 15:613–20. doi: 10.1016/j.acap.2015.08.006

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ballester DA, Filippon AP, Braga C, Andreoli SB. The general practitioner and mental health problems: challenges and strategies for medical education. São Paulo Med J. (2005) 123:72–6. doi: 10.1590/S1516-31802005000200008

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Ross PT, Fitzgerald JT, Ravindranath D, Pfeiffer PN, Singh MK, Weiner SJ, et al. Assessing residents' veteran-centered care skills in the clinical setting. J Grad Med Educ. (2018) 10:279–84. doi: 10.4300/JGME-D-17-00700.1

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Zabar S, Hanley K, Watsula-Morley A, Goldstein J, Altshuler L, Dumorne H, et al. Using unannounced standardized patients to explore variation in care for patients with depression. J Grad Med Educ. (2018) 10:285–91. doi: 10.4300/JGME-D-17-00736.1

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Hodges B, Inch C, Silver I. Improving the psychiatric knowledge, skills, and attitudes of primary care physicians, 1950-2000: a review. Am J Psychiatry. (2001) 158:1579–86. doi: 10.1176/appi.ajp.158.10.1579

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Roter DL, Cole KA, Kern DE, Barker LR, Grayson M. An evaluation of residency training in interviewing skills and the psychosocial domain of medical practice. J Gen Intern Med. (1990) 5:347–54. doi: 10.1007/BF02600404

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Dubosh NM, Hall MM, Novack V, Shafat T, Shapiro NI, Ullman EA. A multimodal curriculum with patient feedback to improve medical student communication: pilot study. West J Emerg Med. (2020) 21:115–21. doi: 10.5811/westjem.2018.11.44318

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Vandekieft GK. Breaking bad news. Am Fam Physician. (2001) 64:1975–9.

Google Scholar

26. Freed GL, Dunham KM, Switalski KE, Jones Jr MD, McGuinness GA. Recently trained general pediatricians: perspectives on residency training and scope of practice. Pediatrics. (2009) 123:S38–43. doi: 10.1542/peds.2008-1578J

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Langewitz WA, Eich P, Kiss A, Wossmer B. Improving communication skills—a randomized controlled behaviorally oriented intervention study for residents in internal medicine. Psychosom Med. (1998) 60:268–78. doi: 10.1097/00006842-199805000-00009

CrossRef Full Text | Google Scholar

28. O'Sullivan P, Chaos S, Russell M, Levine S, Fabiny A. Development and implementation of an objective structured clinical examination to provide formative feedback on communication and interpersonal skills in geriatric training. J Am Geriatr Soc. (2008) 56:1730–5. doi: 10.1111/j.1532-5415.2008.01860.x

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Fujimori M, Shirai Y, Kubota K, Katsumata N, Uchitomi Y. Effect of communication skills training program for oncologists based on patient preferences for communication when receiving bad news: a randomized control trial. J Clin Oncol. (2014) 32:2166–72. doi: 10.1200/JCO.2013.51.2756

CrossRef Full Text | Google Scholar

30. Ditton-Phare P, Sandhu H, Kelly B, Kissane D, Loughland C. Pilot evaluation of a communication skills training program for psychiatry residents using standardized patient assessment. Acad Psychiatry. (2016) 40:768–75. doi: 10.1007/s40596-016-0560-9

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Zuckerbrot RA, Cheung A, Jensen PS, Stein REK, Laraque D. GLAD-PC STEERING GROUP. Guidelines for Adolescent Depression in Primary Care (GLAD-PC): Part I. Practice preparation, identification, assessment, and initial management. Pediatrics. (2018) 141:e20174081. doi: 10.1542/peds.2017-4081

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Horowitz LM, Bridge JA, Teach SJ, Ballard E, Klima J, Rosenstein DL, et al. Ask suicide-screening questions (ASQ): a brief instrument for the pediatric emergency department. Arch Pediatr Adolesc Med. (2012) 166:1170–6. doi: 10.1001/archpediatrics.2012.1276

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Jensen PS, Kratochvil C, Laraque DL, Stein REK. Basic science-guided coaching of primary care providers, in Scientific Proceedings of the Annual Meetings, American Academy of Child & Adolescent Psychiatry. Toronto: American Academy of Child & Adolescent Psychiatry (2011).

34. Bridgemohan C, Bauer NS, Nielson BA, DeBattista A, Ruch-Ross HS, Paul LB, et al. A workforce survey on developmental-behavioral pediatrics. Pediatrics. (2018) 141:e20172164. doi: 10.1542/peds.2017-2164

CrossRef Full Text | Google Scholar

35. Martini R, Hilt R, Marx L, Chenven M, Naylor M, Sarvet B, et al. Best Principles for Integration of Child Psychiatry into the Pediatric Health Home. Am Acad Child Adol Psychiatry. (2012).

36. International association for child and adolescent psychiatry and allied professions. Declaration of the consortium for global infant, child and adolescent mental health. Available online at: https://iacapap.org/content/uploads/Declaration_GlobalInfant.pdf (accessed April 28, 2021).

37. Prince M. Does active learning work? A review of the research. J Engr Educ. (2004) 93:223–31. doi: 10.1002/j.2168-9830.2004.tb00809.x

CrossRef Full Text | Google Scholar

38. Issenberg SB, Mcgaghie WC, Petrusa ER, Gordon DL, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. (2005) 27:10–28. doi: 10.1080/01421590500046924

PubMed Abstract | CrossRef Full Text

39. Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. (2008) 42:128–33. doi: 10.1111/j.1365-2923.2007.02974.x

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Harris Jr JM, Sun H. A randomized trial of two e-learning strategies for teaching substance abuse management skills to physicians. Acad Med. (2013) 88:1357–62. doi: 10.1097/ACM.0b013e31829e7ec6

PubMed Abstract | CrossRef Full Text | Google Scholar

41. O'Brien R, Goldberg SE, Pilnick A, Beeke S, Schneider J, Sartain K, et al. The VOICE study - a before and after study of a dementia communication skills training course. PLoS ONE. (2018) 13. doi: 10.1371/journal.pone.0198567

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Ahn J, Martin SK, Farnan JM, Fromme HB. The graduate medical education scholars track: developing residents as clinician-educators during clinical training via a longitudinal, multimodal, and multidisciplinary track. Acad Med. (2018) 93:214–9. doi: 10.1097/ACM.0000000000001815

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Baccolini V, D'Egidio V, de Soccio P, Migliara G, Massimi A, Alessandri F, et al. Effectiveness over time of a multimodal intervention to improve compliance with standard hygiene precautions in an intensive care unit of a large teaching hospital. Antimicrob Resist Infect Control. (2019) 8:92. doi: 10.1186/s13756-019-0544-0

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Duff B, Massey D, Gooch R, Wallis M. The impact of a multimodal education strategy (the DeTER program) on nurses' recognition and response to deteriorating patients. Nurse Educ Pract. (2018) 31:130–5. doi: 10.1016/j.nepr.2018.05.011

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Ghezeljeh TN, Abbasnejad Z, Rafii F, Haghani H. Effect of a multimodal training program on traditional lecture method on nurses' hand hygiene knowledge, belief, and practice: a brief report. Am J Infect. (2015) 43:762–4. doi: 10.1016/j.ajic.2015.03.018

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Lorio AK, Gore JB, Warthen L, Housley SN, Burgess EO. Teaching demetia care to physical therapy doctoral students: a multimodal experiential learning approach. Gerontol Geriatr Educ. (2017) 38:313–24. doi: 10.1080/02701960.2015.1115979

CrossRef Full Text | Google Scholar

47. Kawamura A, Mylopoulos M, Orsino A, Jimenez E, McNaughton N. Promoting the development of adaptive expertise: exploring a simulation model for sharing a diagnosis of autism with parents. Acad Med. (2016) 91:1576–81. doi: 10.1097/ACM.0000000000001246

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Blashki GA, Piterman L, Meadows GN, Clarke DM, Prabaharan V, Gunn JM, et al. Impact of an educational intervention on general practitioners' skills in cognitive behavioural strategies: a randomised controlled trial. Med J Aust. (2008) 188:S129–32. doi: 10.5694/j.1326-5377.2008.tb01876.x

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Detering K, Silvester W, Corke C, Milnes S, Fullam R, Lewis V, et al. Teaching general practitioners and doctors-in-training to discuss advance care planning: evaluation of a brief multimodality education programme. BMJ Support Palliat. (2014) 4:313–21. doi: 10.1136/bmjspcare-2013-000450

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Douglas SR, Vides de Andrade AR, Boyd S, Leslie M, Webb L, Davis L, et al. Communication training improves patient-centered provider behavior and screening for soldiers' mental health concerns. Patient Educ Couns. (2016) 99:1203–12. doi: 10.1016/j.pec.2016.01.018

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Louie E, Giannopoulos V, Baillie A, Uribe G, Byrne S, Deady M, et al. Translating evidence-based practice for managing comorbid substance use and mental illness using a multimodal training package. J Dual Diagn. (2018) 14:111–9. doi: 10.1080/15504263.2018.1437496

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Clapper TC. Beyond Knowles: what those conducting simulation need to know about adult learning theory. Clin Simul Nurs. (2010) 6:e7–14. doi: 10.1016/j.ecns.2009.07.003

CrossRef Full Text | Google Scholar

53. Taylor DC, Hamdy H. Adult learning theories: implications for learning and teaching in medical education: AMEE Guide No. 83. Med Teach. (2013) 35:e1561–72. doi: 10.3109/0142159X.2013.828153

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. (2009) 42:377–81. doi: 10.1016/j.jbi.2008.08.010

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Harris PA, Taylor R, Minor BL, Elliot V, Fernandez M, O'Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. (2019) 95:103208. doi: 10.1016/j.jbi.2019.103208

PubMed Abstract | CrossRef Full Text | Google Scholar

56. IBM Corp. IBM SPSS Statistics for Windows. Version 24.0. Armonk, NY: IBM Corp (2016).

57. De Leo D, Draper BM, Snowdon J, Kolves K. Contacts with health professionals before suicide: missed opportunities for prevention? Compr Psychiatry. (2013) 54:1117–23. doi: 10.1016/j.comppsych.2013.05.007

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Lehmann R, Hanebeck B, Oberle S, Simon A, Choukair D, Tönshoff B, et al. Virtual patients in continuing medical education and residency training: a pilot project for acceptance analysis in the framework of a residency revision course in pediatrics. GMS J Med Educ. (2015) 32:9. doi: 10.3205/zma000993

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Dudas R, Colbert-Getz J, Balighian E, Cooke D, Golden W, Khan S, et al. Evaluation of a simulation-based pediatric clinical skills curriculum for medical students. Simul Healthcare. (2014) 9:9. doi: 10.1097/SIH.0b013e3182a89154

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Brown R, Doonan S, Shellenberger S. Using children as simulated patients in communication training for residents and medical students: a pilot program. Acad Med. (2005) 80:1114–20. doi: 10.1097/00001888-200512000-00010

PubMed Abstract | CrossRef Full Text | Google Scholar