Thematic analyses of participant survey responses following dermatology ECHO programs with dermoscopy: Practical tips and lessons learned

Introduction: Skin cancer is a major public health concern in the United States, reflecting approximately one in every three cancer diagnoses. Despite the high incidence of skin cancer, access to dermatologists is limited, especially in rural areas. Primary care physicians play a pivotal role in the evaluation of skin conditions, but dermatology training gaps exist in primary care training programs.

Objectives: This study examines the use of the Project ECHO (Extension for Community Healthcare Outcomes) knowledge-sharing framework to provide dermoscopy and skin cancer detection training to primary care providers (PCPs).

Methods: Responses to surveys administered to participants in two separate dermoscopy-focused Project ECHO courses were analyzed. Survey responses were collected over a 4-year period for the two courses, which were delivered in Maine and Texas. Thematic analysis of the qualitative data was performed, revealing codes and subcodes that indicated several overall trends.

Results: Overall, most respondents indicated the ECHO sessions to be helpful, reporting an increase in confidence and knowledge in dermoscopy. Other codes reflected a positive reception of the learning materials and teaching styles. Furthermore, participant survey analyses highlighted areas of improvement for future ECHO course sessions.

Conclusions: This thematic analysis of Project ECHO courses in dermatology with dermoscopy demonstrates the feasibility of using virtual educational platforms to effectively teach PCPs about dermoscopy and skin cancer, with high levels of participant satisfaction. The need to keeping the educational sessions brief, avoid scheduling sessions on high-volume patient care days, and provide a means for participants to obtain hands-on training in the operation of a dermatoscope were among the top lessons learned.

Introduction

The World Health Organization (WHO) estimates that one in every three cancers diagnosed is skin cancer (1). While examinations to detect skin cancer are noted to be among the most effective screening interventions in medicine, skin cancer incidence continues to increase (2–4). Performing skin examinations is an integral part of a dermatologist's practice, yet many regions of the United States have no dermatologists (5). In these regions, primary care providers (PCPs) may be a critical resource in the diagnosis of skin cancers. However, as many as 89% of PCPs cite a lack of dermatologic training as a barrier to performing skin examinations (6).

Two distinct yet quite similar educational interventions were developed to combat this barrier by providing PCPs with foundational skin cancer detection training. The two interventions were similar as they both taught an algorithm-based approach to dermoscopy (7). A dermatoscope is a low-cost hand-held medical instrument that pairs a 10× magnifying lens with a polarized light source. With appropriate training, use of a dermatoscope reduces the rate of both false positives and false negatives in skin cancer detection examinations (8).

Both training interventions used the evidence-based Project ECHO (Extension for Community Healthcare Outcomes) knowledge-sharing framework (9). The courses were distinct in their target geographical areas and end-users. Course “A” was delivered throughout the state of Maine, targeting practicing PCPs in rural counties. Highlighting these rural areas was a key component of this study, given that Maine has fewer than 3 dermatologists per 100,000 people (5). This ratio is among the lowest in the United States, and thus the importance of dispensing this knowledge to PCPs in the state is particularly high. Course “B” was delivered to resident physicians with the Texas Tech University Health Sciences Center Department of Family and Community Medicine in El Paso, TX.

In this study, we conducted a thematic analysis of 474 course-specific survey responses collected over the course of 4 years to determine both the strengths of the courses and areas for possible improvement. In this article, we also summarize best practices, practical tips, and lessons learned that may benefit others developing interactive virtual educational interventions for dermoscopy and skin cancer.

Methods

Both interventions received Institutional Review Board approval, and participants reviewed a statement of consent and agreed to participate in the study. Each participant had the opportunity to opt out of data collection at any time. We then extracted data on programmatic outcomes collected from participants in the two Project ECHO telementoring educational interventions.

Responses were collected in the form of a survey with open- and closed-ended questions; however, this analysis included only the responses to open-ended survey questions. Responses were collected from September 2019 to May 2022. The following data fields were included, addressing qualitative aspects of each participant's experience as outcomes: identifying strengths and weaknesses of each intervention; defining how the participant would change their clinical practices based on exposure to the intervention; and explaining why they participated in the intervention sessions. Participants were also invited to share general comments on the intervention.

All participants were sent a URL via email that they could use to access the survey at the end of the ECHO session. Response rates were assessed to ensure validity. Participants included resident and attending physicians from academic and community health centers, nurse practitioners, and physician assistants who signed up to participate in the Project ECHO intervention. We included all pertinent responses from participants in each of the sessions in the analysis. Responses were analyzed using a descriptive approach to inquiry, identifying themes and patterns from the data rather than creating codes a priori.

We used thematic analysis to identify and investigate trends in the data collected from 2019 to 2022 on the programmatic outcomes of the two interventions. Three team members (TB, JP, and AT) independently reviewed and condensed deidentified survey data to identify general codes and subcodes (Figure 1). These initial codes were discussed with the larger group and modified based on feedback, generating a secondary set of filtered codes. The final rounds of data review and coding generated the finalized set of codes and subcodes presented in this paper. The finalized codes were discussed in the larger group to ensure face validity.

FIGURE 1

Figure 1. Flow chart illustrating the stepwise thematic analysis process.

Once the finalized codes were determined, the data were analyzed to determine the participant-level frequency of each thematic element by ECHO site location. Exemplar quotes were identified to further support the validity of the final codes and to best reflect the overall dataset (Table 1). Data were collected using Smartsheets, data analysis was conducted using Microsoft Excel (Version 2206), and figures were created using R (version 4.0.2) with the ggplot2 package (10).

TABLE 1

Table 1. Themes and subthemes for each qualitative survey question.

Results

The themes, subthemes, and frequency of each subtheme are presented in Table 1. Table 2 provides examples of notable responses coded under each subtheme. In aggregate, 592 responses were coded: 98 under Theme 1 (likes), 302 under Theme 2 (dislikes), and 192 under Theme 3 (motivations for participating).

TABLE 2

Table 2. Subtheme exemplar responses.

Results from both site locations for Theme 1 (likes) indicated that the participants’ favorite aspect of the program was the concise nature of the 1-h presentations followed by the course-integrated practice questions and examples. When the responses were separated by location, the concise nature of the presentation was favored in Maine, while the practice questions and examples were favored in Texas. Figure 2 displays the results pertaining to Theme 1 (likes) in graphical form.

FIGURE 2

Figure 2. Results relating to Theme 1 (likes) and Theme 2 (dislikes/areas for improvement) by location.

Questions addressing Theme 2 components (dislikes) elicited a 47% response rate; 51% (n = 155) of these responses indicated no dislikes or areas for improvement. The most commonly identified area for improvement related to a desire for additional content, as solicited by the question, “What additional dermatology topics would you like to see offered?” Technical difficulty was the second-most common dislike, with increased frequency noted during the initial years of the SARS-CoV-2 pandemic. In Maine, 88% (n = 15) of responses mentioning Zoom (Zoom Video Communications, Inc., San Jose, CA) or other technical difficulties occurred in the 2019 and 2020 surveys, and in Texas, all the responses relating to technical difficulties occurred in the 2020–2021 year. Finally, multiple responses indicated that sessions on Mondays, and sessions longer than one hour, were dispreferred. Figure 2 displays the results pertaining to Theme 2 (dislikes) in graphical form.

There was a 78% response rate for the question relating to Theme 3 (motivations for participating). The primary sources of motivation for participating were desire for improvement (33%, n = 64) and desire to gain or increase proficiency with dermoscopy (19%, n = 36). Other popular motivations included a general interest in dermatology (14%, n = 26), desire to learn about the role of PCPs in dermatology (13%, n = 24), and desire to improve patient care (12%, n = 23). Figure 3 displays the results pertaining to Theme 3 (motivations for participating) in graphical form.

FIGURE 3

Figure 3. Results relating to Theme 3 (reasons for attending) by location.

Demographic data were collected for participants in the Maine ECHO programs. In the 2019–2020 ECHO program, there were 63 participants: 78% were PCPs, 9% were NPs or PAs, and 13% were dermatologists or ancillary medical staff. There was an average of 27 participants per session. In the 2020–2021 ECHO program, there were 81 participants: 61% were PCPs, 33% were NPs or PAs, and 6% were dermatologists or ancillary medical staff. There was an average of 30 participants per session. Finally, in the 2021–2022 ECHO program, there were 86 participants: 56% were PCPs, 36% were NPs or PAs, and 8% were dermatologists or ancillary medical staff. There was an average of 33 participants per session. The number of participants and average attendance increased in each consecutive year. Demographic data were not available for the El Paso, Texas ECHO programs.

Discussion

Our data on programmatic outcomes can inform best practices for educational interventions leveraging the Project ECHO telementoring framework. Positive feedback regarding session duration was provided when the sessions were 1 h long, while negative feedback was given when the sessions lasted 2 h or longer. Thus, 1 h is the preferred session duration. Sessions should also be spaced out, potentially monthly, to increase knowledge retention (11). For sessions lasting more than 1 h, breaks at the hour mark are helpful to decrease the fatigue associated with online learning.

Monday was frequently cited as an inconvenient day for a telementoring session. Evidence suggests that healthcare providers’ offices experience the highest demand for appointments and handle the largest volume of patients on Mondays (12, 13). We suggest holding ECHO sessions on Wednesdays, as healthcare offices have been found to see fewer patients on Wednesdays than Mondays, suggesting that PCPs may be less busy (13). If sessions must be conducted on Mondays, recording the session can provide asynchronous access.

Based on the responses under subtheme 1.1 (“Examples and practice questions”), ECHO sessions should include numerous practice questions and examples. Many responses indicated that these practice questions helped clinicians to establish a visible pattern, enabling them to diagnose skin lesions more accurately.

While this qualitative thematic analysis found that course participants value the program for its ability to increase dermoscopy skills and confidence surrounding skin cancer diagnosis, educational interventions do not always translate to changes in practice or improved patient outcomes. However, following nearly 2 years of provision of this Project ECHO dermatology course in Maine, there was a 10% (p < .0001) shift from benign to malignant biopsy results in the primary care setting (14). Furthermore, there was a meaningful reduction in the number of patients needing to undergo biopsy to detect skin cancer (14, 15).

There was a strong desire for additional content, as 22% of participants (n = 96) responded to the question “What additional dermatology topics would you like to see offered?” This question facilitates improvement of the program by highlighting participants’ current knowledge gaps. We recommend including this question on feedback surveys for all ECHO programs, as the topics mentioned can be incorporated into the following year's curriculum, improving participants’ satisfaction with the course and eliminating knowledge gaps.

For reference, the title of (and thus the topic covered in) each ECHO lecture was as follows: Dermoscopy Basics: Benign Skin Growths; Dermoscopy Basics: Skin Cancer; Basal Cell Carcinoma: Building on Triage Amalgamated Dermoscopic Algorithm (TADA); Nevus vs. Melanoma; Demystifying Skin Biopsies; Psoriasis and Scaly Rashes; Common Skin Infections; Dermatopharmocology; and Dermoscopy Recap.

The lack of responses indicating technical and Zoom-related difficulties in 2021 and 2022 suggests that the use of a virtual setting for ECHO sessions is becoming more feasible and efficient. The virtual setting has multiple advantages, including the ability to teach PCPs from a distance. However, it should be noted that the lack of hands-on experience associated with virtual learning was repeatedly mentioned by respondents. We recommend combating this issue by partnering with a PCP site champion. The Maine ECHO established site champions at each of its ECHO sites. These site champions received training on operation of a dermatoscope, and all of their clinics were equipped with dermatoscopes. Site champions stimulate engagement in the ECHO program and offer in-office, hands-on practice, allowing participants to refine their skills and better achieve the desired program outcomes.

An understanding of participants’ motivations for enrollment prior to conducting the telementoring sessions could be of value; thus, we recommend asking the question “What are your primary reasons for participating in this training?” before the program starts via a precourse survey. With an understanding of the respondents’ motives for participating, the course leaders can better orient their teaching style and the session content toward the participants’ desires, which would likely result in a further increase in satisfaction with the course.

In our study, subthemes 3.4 (“Learning more about the role of a PCP in dermatology”) and 3.5 (“Improve patient care”) are not only applicable to dermoscopy but also generalizable to all ECHO programs in dermatology. As a result, we recommend emphasizing the role of PCPs in dermatology in the delivery of such sessions, such as when to refer to a dermatologist and how to improve patient care.

In summary, our analysis demonstrates high engagement and satisfaction with virtual dermoscopy training delivered via Project ECHO. By incorporating our recommendations, future ECHO programs might achieve improved levels of satisfaction and outcomes. This platform is easily generalizable to all dermatologic topics, and if implemented more widely, Project ECHO may help address the need for more dermatologic care (specifically, skin cancer detection) and training in the primary care setting.

Conclusion

As the need for dermatologic care throughout the country continues to increase, the delivery of virtual educational interventions offers a method to combat gaps in dermatology training. This analysis found that dermoscopy and skin cancer training delivered via the Project ECHO platform is well received. Optimizing partnerships between PCPs and dermatologists has the potential to allow for the transfer of knowledge, rather than referral of patients, for many common skin conditions, including skin cancer.

Data availability statement

The raw data supporting the conclusions of this article may be made available by the authors on a case-by-case basis. Requests to access the datasets should be directed to Elizabeth V. Seiverling, vseiverling@gmail.com.

Ethics statement

The studies involving human participants were reviewed and approved by The University of Texas MD Anderson Cancer Center IRB 19-0536 And Tufts University School of Medicine IRB 1344302-1. The participants provided their written informed consent to participate in this study.

Author contributions

KN, ES, and PC contributed to the conception and design of the study. TB, JP, and AT organized and coded the qualitative data. HS performed the statistical analysis and created the figures. TB and JP wrote the first draft of the manuscript. AT wrote sections of the manuscript. All authors contributed to the article and approved the submitted version.

Acknowledgments

The authors acknowledge Dr. Gerardo Vasquez and Dr. Sarah Sepulveda from Texas Tech University Health Sciences Center, Department of Family and Community Medicine, El Paso, TX, and Dr. Valerie Donahue and Dr. Matthew Turnquist from MaineHealth, Department of Family Medicine, Portland, ME, for their instrumental roles in facilitating the ECHO sessions taught in their respective regions.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. World Health Organization (WHO). Radiation: Ultraviolet (UV) radiation and skin cancer (2017). https://www.who.int/news-room/questions-and-answers/item/radiation-ultraviolet-(uv)-radiation-and-skin-cancer (Accessed June 25, 2022).

2. Moscarella E, Tion I, Zalaudek I, Lallas A, Longo C, Lombardi M, et al. Both short-term and long-term dermoscopy monitoring is useful in detecting melanoma in patients with multiple atypical nevi. J Eur Acad Dermatol Venereol. (2016) 31(2):247–51. doi: 10.1111/jdv.13840

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Losina E, Walensky R, Geller A, Beddingfield F, Wolf L, Gilchrest B, et al. Visual screening for malignant melanoma: a cost-effectiveness analysis. Arch Dermatol. (2007) 143:21–8. doi: 10.1011/archderm.143.1.21

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Godar D. Worldwide increasing incidences of cutaneous malignant melanoma. J Skin Cancer. (2011) 2011:858425. doi: 10.1155/2011/858425

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Glazer A, Farber A, Winkelmann R, Rigel D. Analysis of trends in geographic distribution and density of US dermatologists. JAMA Dermatol. (2017) 153:322–5. doi: 10.1001/jamadermatol.2016.5411

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Kirsner R, Muhkerjee S, Federman D. Skin cancer screening in primary care: prevalence and barriers. J Am Acad Dermatol. (1999) 41:564–6. PMID: 10495376

PubMed Abstract | Google Scholar

7. Seiverling E, Ahrns H, Greene A, Butt M, Yelamos O, Dusza S, et al. Teaching benign skin lesions as a strategy to improve the triage amalgamated dermoscopic algorithm (TADA). J Am Board Fam Med. (2019) 32:96–102. doi: 10.3122/jabfm.2019.01.180049

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Dinnes J, Deeks J, Chuchu N, di Ruffano L, Matin R, Thomson D, et al. Dermoscopy, with and without visual inspection, for diagnosing melanoma in adults. Cochrane Database Syst Rev. (2018) 12:55–60. doi: 10.1002/14651858.CD011902.pub2

CrossRef Full Text | Google Scholar

9. Arora S, Thornton K, Murata G, Deming P, Kalishman S, Dion D, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. (2011) 2:199–207. doi: 10.1056/NEJMoa1009370

CrossRef Full Text | Google Scholar

10. Wickham H. Ggplot2: Elegant graphics for data analysis. Version 3.3.3. New York: Springer (2009).

11. Seiverling E, Li D, Stevens K, Cyr P, Dorr G, Ahrns HV, et al. Distance learning and spaced review to complement dermoscopy training for primary care. Dermatol Pract Concept. (2021) 11(2):2–5. doi: 10.5826/dpc.1102a30

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Fishbacher C, Robertson R. Patients’ difficulties in obtaining appointments—a general practice audit. J R Coll Gen Pract. (1986) 36(287):282–4. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1960568/3761252

PubMed Abstract | Google Scholar

13. Johannessen K, Alexandersen N. Improving accessibility for outpatients in specialist clinics: reducing long waiting times and waiting lists with a simple analytic approach. BMC Health Serv Res. (2018) 18(1):827. doi: 10.1186/s12913-018-3635-3

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Seiverling EV, Stevens K, Dorr G, Prentiss MA, Stoddard H, Houk L, et al. Impact of multimodal dermatoscopy training on skin biopsies by primary care providers. J Am Acad Dermatol. (2022) 87(5):1119–21. doi: 10.1016/j.jaad.2022.01.027

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Seiverling E, Prentiss M, Stevens K, Stoddard H, Cyr P, Ahrns H. Impact of dermoscopy training for primary care practitioners on number needed to biopsy to detect melanoma. PRiMER. (2023) 7:1–3. doi: 10.22454/PRiMER.2023.276659

PubMed Abstract | CrossRef Full Text | Google Scholar