AUTHOR=Tan Darren Ngiap Hao , Tham Yih-Chung , Koh Victor , Loon Seng Chee , Aquino Maria Cecilia , Lun Katherine , Cheng Ching-Yu , Ngiam Kee Yuan , Tan Marcus 

TITLE=Evaluating Chatbot responses to patient questions in the field of glaucoma

JOURNAL=Frontiers in Medicine

VOLUME=11

YEAR=2024

URL=https://www.frontiersin.org/journals/medicine/articles/10.3389/fmed.2024.1359073

DOI=10.3389/fmed.2024.1359073

ISSN=2296-858X

ABSTRACT=<sec id="sec1"><title>Objective</title><p>The aim of this study was to evaluate the accuracy, comprehensiveness, and safety of a publicly available large language model (LLM)—ChatGPT in the sub-domain of glaucoma.</p></sec><sec id="sec2"><title>Design</title><p>Evaluation of diagnostic test or technology.</p></sec><sec id="sec3"><title>Subjects, participants, and/or controls</title><p>We seek to evaluate the responses of an artificial intelligence chatbot ChatGPT (version GPT-3.5, OpenAI).</p></sec><sec id="sec4"><title>Methods, intervention, or testing</title><p>We curated 24 clinically relevant questions in the domain of glaucoma. The questions spanned four categories: pertaining to diagnosis, treatment, surgeries, and ocular emergencies. Each question was posed to the LLM and the responses obtained were graded by an expert grader panel of three glaucoma specialists with combined experience of more than 30 years in the field. For responses which performed poorly, the LLM was further prompted to self-correct. The subsequent responses were then re-evaluated by the expert panel.</p></sec><sec id="sec5"><title>Main outcome measures</title><p>Accuracy, comprehensiveness, and safety of the responses of a public domain LLM.</p></sec><sec id="sec6"><title>Results</title><p>There were a total of 24 questions and three expert graders with a total number of responses of <italic>n</italic> = 72. The scores were ranked from 1 to 4, where 4 represents the best score with a complete and accurate response. The mean score of the expert panel was 3.29 with a standard deviation of 0.484. Out of the 24 question-response pairs, seven (29.2%) of them had a mean inter-grader score of 3 or less. The mean score of the original seven question-response pairs was 2.96 which rose to 3.58 after an opportunity to self-correct (z-score − 3.27, <italic>p</italic> = 0.001, Mann–Whitney U). The seven out of 24 question-response pairs which performed poorly were given a chance to self-correct. After self-correction, the proportion of responses obtaining a full score increased from 22/72 (30.6%) to 12/21 (57.1%), (<italic>p</italic> = 0.026, χ<sup>2</sup> test).</p></sec><sec id="sec7"><title>Conclusion</title><p>LLMs show great promise in the realm of glaucoma with additional capabilities of self-correction. The application of LLMs in glaucoma is still in its infancy, and still requires further research and validation.</p></sec>