AUTHOR=Cohen Yoav 

TITLE=Estimating the Intra-Rater Reliability of Essay Raters

JOURNAL=Frontiers in Education

VOLUME=2

YEAR=2017

URL=https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2017.00049

DOI=10.3389/feduc.2017.00049

ISSN=2504-284X

ABSTRACT=<p>The intra-rater reliability in rating essays is usually indexed by the inter-rater correlation. We suggest an alternative method for estimating intra-rater reliability, in the framework of classical test theory, by using the dis-attenuation formula for inter-test correlations. The validity of the method is demonstrated by extensive simulations, and by applying it to an empirical dataset. It is recommended to use this estimation method whenever the emphasis is not on the average intra-reliability of a group of raters, but when the intra-rater reliability of a specific rater is of interest, e.g., when the error-variance component of the scores is of interest in order to estimate true scores.</p>