Evaluating the Performance of ChatGPT on Patient Consultation and Image-based Preliminary Diagnosis in Thyroid Eye Disease

Yuqing Chen ^1* Yue Wang ^1,2 Shuo Yang ^1,2 Chengcheng Zeng ¹ Yingwei Xie ³ Ya Shen ¹ Li Jian ¹ Xiao Huang ^1* Ruili Wei ^1*

• ¹ Shanghai Changzheng Hospital, Huangpu, Shanghai Municipality, China
• ² Second Military Medical University, Shanghai, China
• ³ Department of Urology, Beijing Tongren Hospital, Capital Medical University, Beijing, China

The emergence of Large Language Model Chatbots, such as ChatGPT, presents great promise for enhancing healthcare practices. Online consultation, accurate pre-diagnosis, and clinical efforts are fundamental to the patient-oriented management system.Objective: This cross-sectional study aims to evaluate the performance of ChatGPT under inquiries across ophthalmic domains, focusing on Thyroid Eye Disease (TED) consultation and image-based preliminary diagnosis in a non-English language.Methods: We sourced the frequently consulted clinical questions from a published reference based on patient consultation data, A Comprehensive Collection of Thyroid Eye Disease Knowledge, and collected facial and Computed Tomography (CT) images from 16 patients with a definitive diagnosis as TED. Inquiries on TED consultation and preliminary diagnosis were posed to ChatGPT from May 18 to 30, 2024, using a new chat for each question. Responses to questions from ChatGPT-4, 4o, along with those from an experienced ocular professor were compiled into three questionnaires participated by patients and ophthalmologists, evaluating the quality in four dimensions: accuracy, comprehensiveness, conciseness, and satisfaction. The preliminary diagnosis of TED was defined as accurate and differences in the accuracy rate were further calculated.Results: For common consultation questions about TED, ChatGPT-4o performed better in delivering a substantial amount of accurate information with logical consistency, adhering to a structured format of disease definition, detailed sections, and summarized conclusions. Notably, the answers generated by ChatGPT-4o were rated high, surpassing that of ChatGPT-4 and the professor, with accuracy (4.33 [0.69]), comprehensiveness (4.17 [0.75]), conciseness (4.12 [0.77]), and satisfaction (4.28 [0.70]). The evaluators' characteristics, answer variables, and other quality scores correlated with overall satisfaction scores. Based on several facial images, ChatGPT-4 twice failed to conduct diagnoses because of lacking characteristic symptoms or complete medical history, while ChatGPT-4o accurately identified the pathologic conditions in 31.25% of cases (95% CI: 11.02%-58.66%). Furthermore, in combination with CT images, ChatGPT-4o performed comparatively to the professor regarding diagnosis accuracy (87.5%, 95% CI 61.65%-98.45%).ChatGPT-4o excelled in comprehensive and satisfactory patient consultation and imaging interpretation, presenting the potential to increase clinical practice efficiency. However, limitations regarding disinformation management and legal permissions remain significant concerns, requiring further exploration in clinical practice.