Skip to main content

ORIGINAL RESEARCH article

Front. Artif. Intell.
Sec. Natural Language Processing
Volume 8 - 2025 | doi: 10.3389/frai.2025.1398885

From Llama to Language: Prompt-Engineering Allows General-Purpose Artificial Intelligence to Rate Narratives Similar to Expert Psychologists

Provisionally accepted
Barry Dauphin Barry Dauphin 1*Caleb Siefert Caleb Siefert 2*
  • 1 Department of Psychology, University of Detroit Mercy, Detroit, United States
  • 2 University of Michigan–Dearborn, Dearborn, Michigan, United States

The final, formatted version of the article will be published soon.

    Artificial intelligence (AI) has tremendous potential for use in psychology. Among the many applications that may benefit from development of AI applications is narrative-personality assessment. Use of these tools and research methods is notably time-consuming and resource intensive. AI has potential to address these issues in ways that would greatly reduce clinician and researcher burden. Nonetheless, it is unclear if current AI models are sufficiently sophisticated to perform the complex downstream tasks, such as narrative assessment. The purpose of this study is to explore if an expert-refined prompt generation process can enable AI-empowered chatbots to reliably and accurately rate narratives using the Social Cognition and Object Relations scales -Global Rating Method (SCORS-G). Experts generated prompt inputs by engaging in a detailed review of SCORS-G training materials. Prompts were then improved using an systematic process in which experts worked with Llama-2-70b to refine prompts. The utility of the prompts was then tested on two AI-empowered chatbots, ChatGPT-4 (OpenAI, 2023) and CLAUDE-2-100k, that were not used in the prompt refinement process. Results showed that the refined prompts allowed chatbots to reliably rate narratives at the global level, though accuracy varied across subscales. Averaging ratings from two chatbots notably improved reliability for the global score and all subscale scores. Experimentation indicated that expert-refined prompts outperformed basic prompts regarding interrater reliability and absolute agreement with gold standard ratings.Only the expert-refined prompts were able to generate acceptable single-rater interrater reliability estimates. Findings suggest that AI could significantly reduce the time and resource burdens on clinicians and researchers using narrative rating systems like the SCORS-G. Limitations and implications for future research are discussed. Deleted: across a broad spectrum of industries, including research 56 and healthcare 57 Deleted: . These chatbots are pre-trained to utilize Natural 58 Language Processing (NLP), allowing them to interact with users via

    Keywords: Social Cognition and Object Relations Scales, artificial intelligence, narrative research, Psychopathology, Personality, Personality Assessment Despite their validity, d Deleted: , often requiring years much more quickly Note for Barry

    Received: 25 Mar 2024; Accepted: 15 Jan 2025.

    Copyright: © 2025 Dauphin and Siefert. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

    * Correspondence:
    Barry Dauphin, Department of Psychology, University of Detroit Mercy, Detroit, United States
    Caleb Siefert, University of Michigan–Dearborn, Dearborn, 48128, Michigan, United States

    Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.