Jacqueline P. Leighton- 1University of Alberta, Edmonton, AB, Canada
- 2Centre for Research in Applied Measurement and Evaluation, Faculty of Education, University of Alberta, Edmonton, AB, Canada
The collection of think aloud data on critical thinking tasks requires participants, many of whom are postsecondary students, to engage with real-life and potentially controversial topics. Accuracy of verbal reports can be enhanced with clear instructions and by minimizing distracting events. For example, interviewers can minimize external distractions such as ambient noise by holding think aloud sessions in a quiet room. However, internal distractions such as participants’ fears about freely expressing their thoughts about controversial topics may be more difficult for interviewers to address. Although the fear of freely expressing thoughts during think aloud interviews has not been empirically investigated, this needs to change. Large-scale surveys indicate that a sizable portion of postsecondary students report discomfort with expressing their thoughts on some topics. This paper offers a theoretical case for why participants’ fears about voicing thoughts freely and without reprisal during think aloud sessions may not only potentially undermine the truthfulness of verbal reports and validity of inferences, but also the very study of critical thinking. Thus, an empirical case for the freedom to think aloud must be considered.
Freedom to think aloud
Think aloud methods are often used to assess how people think about problem-solving tasks (Ericsson and Simon, 1993; Leighton, 2017). In open and free societies, one of the most important goals in higher or postsecondary education is teaching young people to think critically. Critical thinking can be defined as goal-directed thinking that is based on evidence (Hitchcock, 2024). It is often considered synonymous with higher-level forms of reasoning and problem solving (Leighton et al., 2021; Leighton and Sternberg, 2013). Think aloud methods are increasingly used to investigate critical thinking among students to improve its teaching and assessment (Pan et al., 2023).
Critical thinking does not entail teaching students about what values or morals they should adopt. In open and free societies, we accept a pluralism of values, which means that such values will be diverse for different individuals with different experiences, arising from a host of different sources such as religion, culture, and education. For example, the reasoning used by a person to decide whether to buy an electric vehicle might be evaluated by understanding their goal (i.e., buying a good car), the constraints on the goal (e.g., fixed amount of money), and their values of what it means to have a “good car.” One person might value a good car that is environmentally friendly, and another person might value a good car that is easy on the pocketbook. Critical thinking entails teaching students how to select and assess information to serve whatever goals they have chosen to adopt within their value system – and not prescribing to them what values they should adopt.
To investigate critical thinking in postsecondary students, at least in Western countries, a variety of tasks are used. Tasks can be presented with paper-and-pencil or increasingly in ways that are more life-like such as digital performance-based tasks to incite their reasoning. Digitized performance-based tasks present participants with real-life issues that can involve controversial topics such as the environment, health care, and human rights (Chen et al., 2024). The development of real-life critical thinking tasks is done for obvious reasons – to improve the quality of investigations of critical thinking. Once the tasks have been developed, researchers use a variety of tools to measure the response processes used by individuals to think through the tasks. Aside from think aloud methods, other tools include eye-tracking devices, computer-log files, and brain imaging techniques. The application of these tools requires that we trust that participants are engaging in genuine ways with the tasks, for example, that they will be reasonably motivated to follow task instructions, and truthfully report what they are thinking. In other words, researchers must rely on the basic assumption that participants will reveal themselves – not only how they think but also how they feel as they work through the tasks, which has been found to be associated to what they report (Lutsyk-King and Leighton, n.d.). Researchers must assume that they are witnessing a real performance from participants; one that will allow investigators to trust the data and support valid conclusions about what those data might indicate about how individuals are thinking. However, increasingly, postsecondary students find themselves in learning and social environments where voicing thoughts may be fraught with consequences. What is said, even if unintentionally uttered, could be used to judge not just an answer to a question but also the fundamental character of a person. When voicing thoughts becomes risky, there are bound to be potential consequences to our ability to measure thinking.
This paper offers a perspective on why participants’ fears about voicing thoughts freely and without reprisal during think aloud sessions may not only undermine the truthfulness of verbal reports and validity of inferences, but also the very study of critical thinking. Although the explicit link between fear of expression and think aloud reports has not been investigated empirically, this needs to change. The think aloud method relies on participants being comfortable vocalizing their thoughts in front of an interviewer (e.g., Leighton, 2004, 2017, 2021). If students feel afraid disclosing the contents of their thoughts as they solve a particular task, the data they produce will be biased; and any inferences about the quality of their critical thinking will be invalidated as their thinking and performance during the task will have been under duress.
Fear of expression in higher education
College Pulse and the Foundation for Individual Rights and Expression (FIRE) released their 2025 College Free Speech Ranking Report in US colleges (Stevens, 2024). The 2025 report represents the fifth report in a series of reports that track how students feel about expressing themselves on campuses. A scrutiny of the methods used to collect these data is beyond the scope of this paper but readers who wish to consult the report directly, can do so by visiting the FIRE website (www.thefire.org/research/publications/). Building on previous reports, the 2025 College Free Speech Rankings Report sampled 257 colleges and universities in the US, and approximately 58,000 student participants. In their report, a sizable proportion of students claim that they feel uncomfortable talking about many serious topics on college campuses. For example, in response to the general question of how comfortable students felt expressing disagreement with their professors about a controversial political topic in a written assignment, about 50% of students indicated feeling very or somewhat uncomfortable. Generally, female students report being more uncomfortable expressing their views than male students. For example, only 44% of female students and 50% of male students indicated feeling “very” or “somewhat” comfortable expressing their views on controversial political topics during an in-class discussion. Students who identify as very liberal are more likely than students identifying as somewhat liberal, slightly liberal, or conservative in expressing their views. Although a sizable proportion of students claim to feeling uncomfortable, concerns about self-censorship have improved for students according to the 2025 report. For example, in 2025 only 17% of students reported that they could not express their opinion on a subject several times a week because of concerns about how another student, a professor or the university administration would react. In previous years this percentage has been higher at 20 and 22%.
Fear of expression and the measurement of thinking aloud
As a researcher who studies problem-solving and critical thinking among postsecondary students and employs think aloud methods to do so, it is difficult to ignore findings that indicate that students may be afraid to express how they think. The size of the samples included in these College Free Speech Rankings Reports are also difficult to ignore. These findings are also interesting to consider alongside other publications (e.g., Dummitt and Patterson, 2022; Flynn, 2020; Macfarlane, 2021; Mercer, 2022) that outline similar observations on the curtailment of freedom of expression on college campuses. If fear of expression has taken hold of a sizable portion of student participants, how is this fear associated with what students are willing to vocalize during think aloud interviews about critical thinking topics? This question has not been empirically investigated. However, this needs to change given the importance of thinking freely for the truthfulness of data and the validity of inferences.
The think aloud method is a self-report procedure that is used to gather qualitative data about the response processes that participants use to solve tasks (Ericsson and Simon, 1993; Leighton, 2017). Using a one-to-one interview, participants must vocalize their thoughts as they solve the tasks. Originally, when the method was first developed, the tasks presented to participants involved relatively mundane subject matter, inclusive of psychological puzzles, mathematical and/or science problems. However, in the last two decades, a much wider set of tasks have been used, including critical thinking tasks with real-life content, varying subject matter and levels of complexity (Leighton, 2021; Pan et al., 2023). Real-life content includes environmental issues, medical procedures or anything else that participants might encounter in everyday life and where the thinking they apply is consequential and potentially controversial.
Two mechanisms for biased verbal reports
Two key mechanisms have been identified as potentially leading to biased verbal reports (Ericsson and Simon, 1993; Fox et al., 2011; Leighton, 2017, 2021; Wilson, 1994). The first, reactivity, is the threat that the content of what participants report thinking as they are solving a task will change as a result of being asked to think aloud. For example, participants may become distracted, nervous or even influenced in their thinking by elaborative probes or leading questions from the interviewer (e.g., please explain what you just said). The result is that their reported thoughts are contaminated by the very procedure used to elicit the thoughts. The second, non-veridicality, is the threat that the thoughts participants report using to solve the task will not accurately reflect what they would have used had they not been asked to think aloud. Both threats are often inter-related. When relatively mundane tasks such as psychological puzzles are used in think-aloud studies, investigators may not worry about participants actively distorting what they are thinking because it is unlikely that participants fear judgment. But when tasks are less mundane, approximate real-life and have elements of controversy, participants may feel afraid to be judged negatively for what they say and hence be motivated to distort their thinking.
Fear of expressing thoughts and therefore changing one’s thoughts during a think aloud interview presents the potential for reactivity. Distorting what one reports in an interview presents a case for non-veridicality as the report fails to accurately represent what the person thinks. A theoretical example may help to illustrate how (1) fear of expression might lead to or be associated with (2) reactivity and non-veridicality in verbal reports.
Example of fear of expression triggering reactivity and non-veridicality
Imagine the following: A university professor undertakes a research project to examine critical thinking among postsecondary students. The professor develops a performance-based assessment task that requires student participants to make an evidence-based recommendation about the COVID-19 vaccine to a friend. Using a think-aloud interview, the professor presents the task to participants. Each participant interviewed is shown the available evidence - a research report from the Centers for Disease Control, an article from the Washington Post and an anecdotal account of adverse effects related to the vaccine. The participating student is asked to think aloud as they review the evidence and make a recommendation about whether vaccination is advisable. This is a typical scenario for many think-aloud studies. However, of note is the topic of the task; this is not a math puzzle but, instead, a controversial real-life topic involving COVID-19.
Imagine yourself as a student participating in this interview and being asked to express your thoughts about this topic in front of a professor or research assistant. The controversy around how COVID-19 is viewed has been empirically established as falling along political lines. For example, in a study of close to 1,000 crowdsourced workers, Peng (2022) found that political ideology was associated with COVID-19 vaccine acceptance. Individuals who espoused left-wing authoritarianism were much more likely to accept the vaccine and support vaccine mandates than individuals who identified as libertarian or with right-wing authoritarianism. Peng’s (2022) study included crowdsourced workers, but postsecondary students also find COVID-19 controversial along political lines. When FIRE and College Pulse asked postsecondary students about COVID-19 for their 2022 report, “a greater percentage of conservative students, compared to liberal and moderate students, identified… COVID-19 vaccine mandates…mask mandates… as topics that were difficult to have an open and honest conversation about” (College Pulse and FIRE, 2022, p. 38). In addition, when the author anonymously surveyed almost 200 undergraduate students in a teacher education program about their comfort level with having honest conversations about topics such as COVID-19, political leaning also mattered. In particular, 70% and 62% of students with independent and left-leaning political ideologies, respectively, felt comfortable having open and honest conversations about COVID-19 on campus; however, only 50% of right-leaning students felt the same. Students who are right leaning may express less comfort in discussing certain topics because they know that their views are not shared by many of their professors. This is not surprising. Dummitt and Patterson (2022) published a report outlining that approximately 85% of professors on Canadian campuses identified primarily as somewhat left or very left.
A participating student who is asked to think aloud about a task that requires revealing a minority political value and/or belief may feel incentivized to hide or distort what they think to avoid judgment by an interviewer who is perceived as powerful. The act of hiding or distorting thoughts in the think-aloud interview is the definition of reactivity in verbal reports. A reactive verbal report will be associated with non-veridicality as what has been uttered by the participating student is not representative of what they really think. Alternatively, if participating students cannot figure out quickly enough how to alter their thoughts in response to the critical thinking task, they could simply disengage entirely and produce a minimal number of utterances alongside a sparse verbal report. The upshot of such a situation is noisy data that are lacking in truthfulness about what participants really think.
Discussion and conclusion
Threats to the quality of data from think aloud interviews have been commonly viewed as arising from external distractions such as noise or leading questions from the interviewer (Fox et al., 2011; Leighton, 2017). These continue to be sources of threat of course. However, a new potential threat has emerged. This new threat has more to do with the larger social context and the freedom that participants might feel in expressing their thoughts without reprisal, especially when their thoughts may be at odds with prevailing campus or interviewer norms. Again, the link between fear of expression and its impact on think aloud data has not been empirically investigated. However, the purpose of this perspective is to articulate the theoretical case for such an empirical investigation. Fear of expression may play a significant role in reactivity and non-veridicality of verbal reports especially when the tasks involve topics that can be considered controversial. Think aloud interviews designed to measure critical thinking require the freedom to think; if participants do not experience this freedom, we are certainly not measuring critical thinking.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.
Author contributions
JL: Conceptualization, Investigation, Resources, Validation, Writing – original draft, Writing – review & editing, Funding acquisition, Supervision.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. Preparation of this paper was supported by a grant to the author from the Social Sciences and Humanities Research Council of Canada (SSHRC Grant No. 435–2016-0114). Grantees undertaking such projects are encouraged to express freely their professional judgment. This paper, therefore, does not necessarily represent the positions or the policies of the Canadian government, and no official endorsement should be inferred.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Chen, F., Cui, Y., Lutsyk-King, A., Gao, Y., Liu, X., Cutumisu, M., et al. (2024). Validating a novel digital performance-based assessment of data literacy: psychometric and eye-tracking analyses. Educ. Inf. Technol. 29, 9417–9444. doi: 10.1007/s10639-023-12177-7
College Pulse and FIRE. (2022). 2022-2023 college free speech rankings: what is the state of free speech on America’s college campuses? The Foundation for Individual Rights and Expression. Available at: https://www.thefire.org/research-learn/2022-2023-college-free-speech-rankings (Accessed September 10, 2024).
Dummitt, C., and Patterson, Z. (2022). The viewpoint diversity crisis at Canadian universities: Political homogeneity, self-censorship, and threats to academic freedom. Ottawa: Macdonald-Laurier Institute.
Ericsson, K. A., and Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.
Flynn, J. R. (2020). A book too risky to publish: Free speech and universities. Washington, DC: Academica Press.
Fox, M. C., Ericsson, K. A., and Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychol. Bull. 137, 316–344. doi: 10.1037/a0021663
Hitchcock, D. (2024). “Critical thinking” in The Stanford encyclopedia of philosophy. eds. E. N. Zalta and U. Nodelman.
Leighton, J. P. (2004). Avoiding misconception, misuse, and missed opportunities: the collection of verbal reports in educational achievement testing. Educ. Meas. Issues Pract. 23, 6–15. doi: 10.1111/j.1745-3992.2004.tb00164.x
Leighton, J. P. (2017). Using think-aloud interviews and cognitive labs in educational research. Oxford, UK: Oxford University Press.
Leighton, J. P. (2021). Rethinking think-alouds: the often-problematic collection of response process data. Appl. Meas. Educ. 34, 61–74. doi: 10.1080/08957347.2020.1835911
Leighton, J. P., Cui, Y., and Cutumisu, M. (2021). Key information processes for thinking critically in data-rich environments. Front. Educ. 6:561847. doi: 10.3389/feduc.2021.561847
Leighton, J. P., and Sternberg, R. J. (2013). “Reasoning and problem solving” in Handbook of psychology: Experimental psychology. eds. A. F. Healy, R. W. Proctor, and I. B. Weiner. 2nd ed (Hoboken, NJ: John Wiley & Sons, Inc), 631–659.
Lutsyk-King, A., and Leighton, J. P. (n.d.). The conscious experience of thinking aloud: are positive feelings associated with better think aloud performance. Psychol. Conscious. Theory Res. Pract.
Mercer, M. (2022). In praise of dangerous universities and other essays. Winnipeg: Frontier Center for Public Policy.
Pan, Z., Cui, Y., Leighton, J. P., and Cutumisu, M. (2023). Insights into computational thinking from think-aloud interviews: a systematic review. Appl. Cogn. Psychol. 37, 71–95. doi: 10.1002/acp.4029
Peng, Y. (2022). Politics of COVID-19 vaccine mandates: left/right-wing authoritarianism, social dominance orientation, and libertarianism. Personal. Individ. Differ. 194:111661. doi: 10.1016/j.paid.2022.111661
Stevens, S.T. (2024). 2025 college free speech rankings: what is the state of free speech on America’s college campuses? The Foundation for Individual Rights and Expression. Available at: https://www.thefire.org/research-learn/2025-college-free-speech-rankings (Accessed September 1, 2024).
Keywords: think alouds, protocol analysis, freedom of expression, methodology, validity
Citation: Leighton JP (2024) Freedom to think aloud. Front. Educ. 9:1518075. doi: 10.3389/feduc.2024.1518075
Edited by:
Margaret Grogan, Chapman University, United StatesReviewed by:
Nélia Maria Pontes Amado, University of Algarve, PortugalCopyright © 2024 Leighton. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jacqueline P. Leighton, amFjcXVlbGluZS5sZWlnaHRvbkB1YWxiZXJ0YS5jYQ==