94% of researchers rate our articles as excellent or good
Learn more about the work of our research integrity team to safeguard the quality of each article we publish.
Find out more
ORIGINAL RESEARCH article
Front. Digit. Health
Sec. Health Technology Implementation
Volume 7 - 2025 | doi: 10.3389/fdgth.2025.1530442
A Comparative Analysis of Large Language Models on Clinical Questions for Autoimmune Diseases
Provisionally accepted
Jing Chen 1 Juntao Ma 2 Jie Yu 3 Weiming Zhang 3 Yijia Zhu 3 Jiawei Feng 3
Linyu Geng 1 Xianchi Dong 4
Huayong Zhang 4
Yuxin Chen 2*
Ming-zhe Ning 2*- 1 State Key Laboratory of Pharmaceutical Biotechnology, School of Life Sciences, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing University,, Nanjing, China
- 2 Department of Laboratory Medicine, Nanjing Drum Tower Hospital Clinical College of Nanjing Medical University, Nanjing, Jiangsu, China, Nanjing, Liaoning Province, China
- 3 Department of Clinic Nutrition, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing, Jiangsu, China, Nanjing, Liaoning Province, China
- 4 State Key Laboratory of Pharmaceutical Biotechnology, School of Life Sciences, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing University, Nanjing, China
Background: Artificial intelligence (AI) has made great strides. To explore the potential of Large Language Models (LLMs) in providing medical services to patients and assisting physicians in clinical practice, our study evaluated the performance in delivering clinical questions related to autoimmune diseases.Methods: 46 questions related to autoimmune diseases were input into ChatGPT 3.5, ChatGPT 4.0, and Gemini. The responses were then evaluated by rheumatologists based on five quality dimensions: relevance, correctness, completeness, helpfulness, and safety. Simultaneously, the responses were assessed by laboratory specialists across six medical fields: concept, clinical features, report interpretation, diagnosis, prevention and treatment, and prognosis. Finally, statistical analysis and comparisons were performed on the performance of the three chatbots in the five quality dimensions and six medical fields.Results: ChatGPT 4.0 outperformed both ChatGPT 3.5 and Gemini across all five quality dimensions, with an average score of 199.8 ± 10.4, significantly higher than ChatGPT 3.5 (175.7 ± 16.6) and Gemini (179.1 ± 11.8) (p = 0.009 and p = 0.001, respectively). The average performance differences between ChatGPT 3.5 and Gemini across these five dimensions were not statistically significant. Specifically, ChatGPT 4.0 demonstrated superior performance in relevance (p < 0.0001, p < 0.0001), completeness (p < 0.0001, p = 0.0006), correctness (p = 0.0001, p = 0.0002), helpfulness (p < 0.0001, p < 0.0001), and safety (p < 0.0001, p = 0.0025) compared to both ChatGPT 3.5 and Gemini. Furthermore, ChatGPT 4.0 scored significantly higher than both ChatGPT 3.5 and Gemini in medical fields such as report interpretation (p < 0.0001, p = 0.0025), prevention and treatment (p < 0.0001, p = 0.0103), prognosis (p = 0.0458, p = 0.0458).Conclusions: This study demonstrates that ChatGPT 4.0 significantly outperforms ChatGPT 3.5 and Gemini in addressing clinical questions related to autoimmune diseases, showing notable advantages across all five quality dimensions and six clinical domains. These findings further highlight the potential of large language models in enhancing healthcare services.
Keywords: Large language models, Autoimmune Diseases, ChatGPT 4.0, Gemini, clinical questions
Received: 21 Nov 2024; Accepted: 14 Feb 2025.
Copyright: © 2025 Chen, Ma, Yu, Zhang, Zhu, Feng, Geng, Dong, Zhang, Chen and Ning. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence:
Yuxin Chen, Department of Laboratory Medicine, Nanjing Drum Tower Hospital Clinical College of Nanjing Medical University, Nanjing, Jiangsu, China, Nanjing, Liaoning Province, China
Ming-zhe Ning, Department of Laboratory Medicine, Nanjing Drum Tower Hospital Clinical College of Nanjing Medical University, Nanjing, Jiangsu, China, Nanjing, Liaoning Province, China
Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.