AUTHOR=Tosti-Kharas Jennifer , Conley Caryn TITLE=Coding Psychological Constructs in Text Using Mechanical Turk: A Reliable, Accurate, and Efficient Alternative JOURNAL=Frontiers in Psychology VOLUME=7 YEAR=2016 URL=https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.00741 DOI=10.3389/fpsyg.2016.00741 ISSN=1664-1078 ABSTRACT=

In this paper we evaluate how to effectively use the crowdsourcing service, Amazon's Mechanical Turk (MTurk), to content analyze textual data for use in psychological research. MTurk is a marketplace for discrete tasks completed by workers, typically for small amounts of money. MTurk has been used to aid psychological research in general, and content analysis in particular. In the current study, MTurk workers content analyzed personally-written textual data using coding categories previously developed and validated in psychological research. These codes were evaluated for reliability, accuracy, completion time, and cost. Results indicate that MTurk workers categorized textual data with comparable reliability and accuracy to both previously published studies and expert raters. Further, the coding tasks were performed quickly and cheaply. These data suggest that crowdsourced content analysis can help advance psychological research.