Skip to main content

BRIEF RESEARCH REPORT article

Front. Psychol.
Sec. Quantitative Psychology and Measurement
Volume 15 - 2024 | doi: 10.3389/fpsyg.2024.1304870

How Does Item Wording Affect Participants' Responses? Evidence from IRT Analysis

Provisionally accepted
Biao Zeng Biao Zeng 1Minjeong Jeon Minjeong Jeon 2*Hongbo Wen Hongbo Wen 1*
  • 1 Collaborative Innovation Center of Assessment for Basic Education Quality, Beijing Normal University, Beijing, China
  • 2 Department of Education, School of Education and Information Studies, University of California, Los Angeles, Los Angeles, California, United States

The final, formatted version of the article will be published soon.

    Researchers often combine both positively and negatively worded items when constructing Likert scales. This combination, however, can introduce method effects due to the variances in item wording. Although previous studies have tried to quantify these effects using factor analysis on scales with different content, the impact of varied item wording on participants' choices among specific options remains undefined.To address this gap, we utilized four versions of the Undergraduate Learning Burnout (ULB) scale, each characterized by a unique valence of item wording. After collecting responses from 1,131 college students, we employed unidimensional, multidimensional, and Bi-factor Graded Response Models for analysis. The results suggest that the ULB scale supports a unidimensional structure for the learning burnout trait. However, the inclusion of different valences of wording within items introduces additional method factors, explaining a considerable degree of variance. Notably, positively worded items demonstrate greater discriminative power and more effectively counteract the biased outcomes associated with negatively worded items, notably between the "Strongly Disagree" and "Disagree" options. While there were no substantial differences in the overall learning burnout traits among respondents of different scale versions, slight variations were noted in their distributions. The integration of both positive and negative wordings reduced the reliability of the learning burnout trait measurement. Consequently, it is recommended to use exclusively positively worded items and avoid a mix in item wording during scale construction. If a combination is essential, the Bi-Factor IRT model might help segregate the method effects resulting from the wording valence.

    Keywords: Likert scale, Item wording, method effect, positively worded, negatively worded, Bi-factor, Graded response model, IRT

    Received: 30 Sep 2023; Accepted: 02 Sep 2024.

    Copyright: © 2024 Zeng, Jeon and Wen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

    * Correspondence:
    Minjeong Jeon, Department of Education, School of Education and Information Studies, University of California, Los Angeles, Los Angeles, 90095, California, United States
    Hongbo Wen, Collaborative Innovation Center of Assessment for Basic Education Quality, Beijing Normal University, Beijing, China

    Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.