Skip to main content

BRIEF RESEARCH REPORT article

Front. Artif. Intell.
Sec. Medicine and Public Health
Volume 7 - 2024 | doi: 10.3389/frai.2024.1452469

Large Language Model Triaging of Simulated Nephrology Patient Inbox Messages

Provisionally accepted
Justin Pham Justin Pham 1Charat Thongprayoon Charat Thongprayoon 2Jing Miao Jing Miao 2Supawadee Suppadungsuk Supawadee Suppadungsuk 3Priscilla Koirala Priscilla Koirala 4Iasmina M. Craici Iasmina M. Craici 2Wisit Cheungpasitporn Wisit Cheungpasitporn 2*
  • 1 College of Medicine and Science, Mayo Clinic, Rochester, Minnesota, United States
  • 2 Division of Nephrology and Hypertension, Mayo Clinic, Rochester, Michigan, United States
  • 3 Chakri Naruebodindra Medical Institute (CNMI), Faculty of Medicine, Ramathibodi Hospital, Mahidol University, Samutprakarn, Thailand
  • 4 Department of General Internal Medicine, Mayo Clinic, Rochester, Minnesota, United States

The final, formatted version of the article will be published soon.

    Background: Efficient triage of patient communications is crucial for timely medical attention and improved care. This study evaluates ChatGPT's accuracy in categorizing nephrology patient inbox messages, assessing its potential in outpatient settings. Methods: 150 simulated patient inbox messages were created based on cases typically encountered in everyday practice at a nephrology outpatient clinic. These messages were triaged as non-urgent, urgent, and emergent by two nephrologists. The messages were then submitted to ChatGPT-4 for independent triage into the same categories. The inquiry process was performed twice with a two-week period in between. ChatGPT responses were graded as correct (agreement with physicians), overestimation (higher priority), or underestimation (lower priority). Results: In the first trial, ChatGPT correctly triaged 140 (93%) messages, overestimated the priority of 4 messages (3%), and underestimated the priority of 6 messages (4%). In the second trial, it correctly triaged 140 (93%) messages, overestimated the priority of 9 (6%), and underestimated the priority of 1 (1%). The accuracy did not depend on the urgency level of the message (p = 0.19). The internal agreement of ChatGPT responses was 92% with an intra-rater Kappa score of 0.88. Conclusion: ChatGPT-4 demonstrated high accuracy in triaging nephrology patient messages, highlighting the potential for AI-driven triage systems to enhance operational efficiency and improve patient care in outpatient clinics.

    Keywords: Large Language Model, ChatGPT, inbox messages, Triage, Patient Care, Patient communication, artificial intelligence

    Received: 25 Jun 2024; Accepted: 29 Aug 2024.

    Copyright: © 2024 Pham, Thongprayoon, Miao, Suppadungsuk, Koirala, Craici and Cheungpasitporn. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

    * Correspondence: Wisit Cheungpasitporn, Division of Nephrology and Hypertension, Mayo Clinic, Rochester, 55905, Michigan, United States

    Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.