Skip to main content

ORIGINAL RESEARCH article

Front. Educ.
Sec. Digital Learning Innovations
Volume 10 - 2025 | doi: 10.3389/feduc.2025.1506752
This article is part of the Research Topic Emerging Technologies and Digital Innovations: Recent Research and Practices in Technology-enhanced Learning Environments View all 9 articles

Differential effects of GPT-based tools on comprehension of standardized passages

Provisionally accepted
Hudson Etkin Hudson Etkin 1*Kai Etkin Kai Etkin 1*Ryan Carter Ryan Carter 2Camarin Rolle Camarin Rolle 3
  • 1 Los Altos High School, Los Altos, United States
  • 2 San Jose State University, San Jose, California, United States
  • 3 Stanford University, Stanford, California, United States

The final, formatted version of the article will be published soon.

    Due to the rapidly improving capability of large language models such as Generative Pre-trained Transformer models (GPT), artificial intelligence (AI) based tools have entered use in education at scale. However, empirical data are largely lacking on the effects of AI tools on learning. Here, we determine the impact of four GPT-based tools on college-aged participants’ reading comprehension of standardized American College Test (ACT)-derived passages using a randomized cross-over online study (n=195). The four tools studied were AI-generated summaries, AI-generated outlines, a question-and-answer tutor chatbot, and a Socratic discussion chatbot. Consistent with our pre-registered hypotheses, we found a differential effect of AI tools as a function of baseline reading comprehension ability. AI tools significantly improved comprehension in lower performing participants and significantly worsened comprehension in higher performing participants. With respect to specific tools, low performers were most benefited by the Socratic chatbot while high performers were worsened most by the summary tool. These findings suggest that while AI tools have massive potential to enhance learning, blanket implementation may cause unintended harm to higher-performing students, calling for caution and further empirical study by developers and educators.

    Keywords: artificial intelligence, Education, reading comprehension, gpt, AI tutoring systems, AI in Education

    Received: 06 Oct 2024; Accepted: 14 Jan 2025.

    Copyright: © 2025 Etkin, Etkin, Carter and Rolle. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

    * Correspondence:
    Hudson Etkin, Los Altos High School, Los Altos, United States
    Kai Etkin, Los Altos High School, Los Altos, United States

    Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.