Akira Miyake
Nicholas P. Carruth- 1Department of Psychology and Neuroscience, University of Colorado Boulder, Boulder, CO, United States
- 2Department of Psychology, DePaul University, Chicago, IL, United States
Introduction
Job et al. (2010; hereafter JDW) proposed that one's susceptibility to the ego-depletion effect—a performance decrement on a self-control task after performing on another self-control task—depends on one's implicit beliefs or mindset about whether willpower is limited or not (hereafter willpower mindset). Specifically, JDW demonstrated that only individuals holding a limited-willpower mindset exhibit a significant ego-depletion effect, a finding often regarded as a compelling refutation of a dominant resource-based account of the ego-depletion effect, the strength model (Baumeister et al., 2007). Berkman (2020) even called JDW's finding “the biggest bombshell” against the strength model.
Although we agree that the strength model has various theoretical problems (Lurquin and Miyake, 2017), JDW's willpower-mindset account also has major problems. Here, we first describe statistical problems that cast doubts on the robustness of JDW's original finding. We then present a review of existing ego-depletion studies we conducted to assess its replicability. We conclude by discussing additional major challenges surrounding JDW's account.
Statistical problems
The statistical evidence for JDW's original finding (Study 1) is on shaky ground. JDW used a letter-cancelation (e-crossing) task as the depletion task and a Stroop task as the outcome task (N = 60). They analyzed the data using logistic hierarchical linear models (HLMs), with binary-coded accuracy on each Stroop incongruent trial (24 trials) as the dependent variable, and reported a significant interaction between the experimental manipulation (depletion vs. control) and willpower mindset: “t(1433) = 3.88, p < 0.01” (p. 1688).
The reported degrees of freedom (1433), however, suggest that the model was likely misspecified, failing to properly nest trials under participants and thereby treating all 1440 trials—24 trials × 60 subjects—as independent observations (Schimmack, 2016). Their model also included three theoretically unmotivated covariates [trial order, reaction time (RT), and age], which may have increased the odds for false-positive results (Simmons et al., 2011).
JDW's subsequent studies analyzed the data differently, even when Stroop was the outcome task. Job et al. (2013) Experiment 1 (N = 87) used standard regressions, with Stroop interference in RTs (incongruent-congruent differences) as the dependent measure and age and lab rooms as covariates, and demonstrated depletion-reducing benefits of glucose consumption only among individuals holding a limited-willpower mindset. Savani and Job (2017) Study 3 used HLMs (properly nesting trials under subjects), with log-transformed RTs as the dependent measure and age and SES as covariates, and demonstrated a reverse ego-depletion effect (better Stroop performance under depletion) only among participants from India (n = 56) who collectively believed that exerting self-control is energizing (p = 0.045).
Such substantial data-analytic differences across studies raise doubts on the robustness of JDW's original finding, a concern exacerbated by our own failure to replicate the moderation effect in a preregistered study (N = 187; Carruth et al., 2023). To further assess the replicability of JDW's finding, we reviewed the existing literature.
Review of existing literature
We conducted a literature search on Google Scholar in January 2023. We first identified all articles that cited the JDW study and then excluded (a) review/theoretical papers without any data and (b) research papers that did not report any ego-depletion studies. We examined the remaining papers to identify studies that administered JDW's willpower-mindset measure and tested its moderating influence on ego depletion. We included only studies treating willpower mindset as an individual-differences variable; those that experimentally manipulated willpower mindset were excluded (e.g., JDW's Study 2; Vohs et al., 2012, Experiment 1).
Thirteen independent studies (from 11 papers) met our inclusion criteria. We did not conduct a formal meta-analysis because different methods used to analyze the data in these studies (e.g., HLM, standard regressions)—some including covariates—made it difficult to cleanly derive a common effect-size metric for the hypothesized interaction effect without having access to the raw data.
We instead classified the 13 studies into two categories—those that observed moderating influences (Category A) and those that did not (Category B)—and examined whether they differed systematically. For this purpose, we coded the following variables: (a) final sample size; (b) data-analytic method; (c) timing of mindset measurement; (d) the study's primary focus; (e) significance of the overall ego-depletion effect; (f) significance of the moderation effect; (g) preregistration; and (h) data availability. The results are summarized in Table 1.
Table 1. Summary of the existing studies testing the moderating influence of willpower mindset on the ego-depletion effect.
Five studies reported the moderating influence of willpower mindset on ego depletion (Category A). As explained in the table, however, the evidence in 4 of the 5 studies was ambiguous (including JDW's original study). Particularly noteworthy are two studies by Singh and Göritz (2019), both reporting the opposite moderation effect (greater depletion for those holding an unlimited-willpower mindset). Only one study (Chow et al., 2015) seems to have demonstrated the hypothesized moderation effect unambiguously.
Eight studies reported no significant moderation effect (Category B), even though the Category B studies collectively had greater sample sizes (mean N = 834.3; median N = 275.5) than the Category A studies (mean N = 464.5; median N = 96.5). Additionally, the hypothesized moderation effect did not seem to depend on the statistical significance of the overall ego-depletion effect: Although all 5 Category A studies reported some evidence for it, so did the majority of the Category B studies (6 out of 8).
The two categories of studies differed in several intriguing ways, however. First, all 5 studies that used HLMs and properly nested trials under participants failed to observe the moderation effect. Thus, it would be informative to know whether JDW's original moderation effect would still be significant if their data were analyzed with proper nesting.
Second, all Category A studies administered the willpower-mindset measure before the depletion manipulation. Administering willpower-mindset and other self-control-related measures early has the advantage of avoiding questionnaire responses being influenced by the depletion manipulation, but it runs the risk of the questionnaire items influencing participants' subsequent behaviors. Interestingly, all 5 studies that administered the mindset questionnaire at the end and thereby minimized such potential priming influences failed to observe the moderation effect.
Third, testing the moderating influence of willpower mindset on ego depletion was the primary focus in 4 of the 5 Category A studies, but in only 1 of the 8 Category B studies (the other studies focused primarily on the replicability of the ego-depletion effect itself; willpower mindset was included as one of the few potential moderators and did not receive much discussion). One way to interpret this discrepancy is in terms of publication bias: If the study's main purpose is to test the moderation effect, publishing a nonsignificant result may be more difficult, hence resulting in few such studies in Category B.
Finally, almost all Category B studies were preregistered (minimizing the likelihood of p-hacking) and have publicly shared the data (enabling other researchers to verify the results and reanalyze the data), but not for the Category A studies, with one notable exception of the Dang et al. (2021) study. The evidence reported in that preregistered multilab study was ambiguous, however: The moderation effect was significant (p = 0.039) only for their secondary dependent measure (RTs for the antisaccade task), and the effect became nonsignificant when the analyses excluded subjects who likely failed to understand the task requirements and performed the outcome task at chance level.
This brief review is limited in two important ways. First, we did not conduct a formal meta-analysis and hence cannot provide an aggregate effect-size estimate for the target interaction term or quantitative tests of the moderating variables we coded for each study. Second, this review was restricted to published studies. Accordingly, it is important to keep in mind that a meta-analysis that includes unpublished studies may yield different results. It would therefore be worthwhile to conduct such a meta-analysis, especially based on raw data so that effect sizes for the target interaction term can be estimated more equivalently across studies.
Despite these limitations, our review showed that the existing evidence for JDW's original finding is weak at best: No large-sample preregistered study currently exists that yielded robust evidence for the hypothesized moderation effect for the primary dependent measure. The finding that only one of the 13 studies yielded an unambiguous moderation effect in the predicted direction casts seriously questions the replicability of JDW's original finding.
Additional challenges
JDW's willpower-mindset account of ego depletion also faces other challenges. First, their account assumes that the ego-depletion effect is “real” at least among some individuals. However, both meta-analyses (Carter and McCullough, 2014; Carter et al., 2015) and large-scale multilab studies (Hagger et al., 2016; Vohs et al., 2021) have seriously challenged the replicability of the ego-depletion effect itself (Friese et al., 2019). Moreover, self-reports from researchers who have conducted ego-depletion studies revealed an alarming rate of prior engagement in questionable research practices, especially selective reporting of outcomes (Wolff et al., 2018). Thus, it is far from clear whether the ego-depletion effect is real.
Even if this controversial effect is real, its overall effect size would likely be small. Three preregistered multilab studies of ego depletion, for example, yielded effect-size estimates (Cohen's d) of only 0.04 (Hagger et al., 2016), 0.08 (Vohs et al., 2021), and 0.10 (Dang et al., 2021), respectively. Detecting a difference between two independent groups with a 0.80 power requires >3,000 participants when the effect size is 0.10. Given that detecting a statistical moderation effect involving individual-differences variables has been known to be difficult (McClelland and Judd, 1993), replicating JDW's original finding (N = 60) with sufficient power would present a major challenge in future research, requiring much larger sample sizes than used before in ego-depletion studies.
JDW's account is also lacking theoretically (Bertrams, 2020). Although not explicitly stated in the original article, Job's preferred explanation is motivational: Performing a demanding self-control task reduces one's motivation level, especially for those holding a limited-willpower mindset, thus making them more willing to “slack off” on a subsequent task (see Carruth et al., 2023, for further elaboration). Though plausible, this motivational account of ego depletion is not precise enough to allow other researchers to unambiguously derive testable (falsifiable) predictions, a point we also raised against the strength model (Lurquin and Miyake, 2017). Without better specifying under what circumstances individuals with a limited-willpower mindset decide to “slack off,” it is difficult to unambiguously predict whether willpower mindset has a moderating influence on ego depletion in a given study.
Equally challenging is the observation that the existing studies showed three different patterns: (a) stronger depletion for limited-willpower mindset; (b) stronger depletion for unlimited-willpower mindset; and (c) no effect of willpower mindset. Given that JDW's account is unidirectional (limited-willpower mindset → greater depletion effects), it cannot explain the opposite pattern reported by Singh and Göritz (2019). Considering that Clarkson et al. (2016) and Vohs et al. (2012) (Experiment 1) demonstrated that an experimentally induced unlimited-willpower mindset can lead to worse self-control performance under some circumstances, one cannot simply dismiss the Singh and Göritz (2019) finding as a fluke. If (a) and (b) are both real and reflect two different ways in which willpower mindset impacts participants' motivation for self-control, JDW's account clearly needs a further theoretical specification.
One possible way forward is to specify potential underlying moderators and mediators that can help reconcile patterns (a) and (b). In fact, Clarkson et al. (2016) proposed (dis)fluency (subjective ease/difficulty of retrieving or processing information) as the moderator (fluency promotes JDW's original pattern, whereas disfluency reverses it) and perceived mental fatigue as the mediator. It is an intriguing proposal, but, because Clarkson et al. (2016) did not test their account using an ego-depletion paradigm, it is unclear whether this account is readily applicable to the ego-depletion results reviewed here.
We believe that willpower mindset will likely remain a useful concept in self-control research (see Francis and Job, 2018, for a review). However, its application to ego depletion—the very phenomenon whose existence has been questioned—is not as compelling as often portrayed in the literature: The moderating influence of willpower mindset on ego depletion is weak at best, starting with JDW's original study. Until more compelling evidence is provided and a more precise model is developed, JDW's original finding must be treated with great caution.
Author contributions
AM and NC jointly developed the ideas and the plan for the literature review presented in this article. NC conducted the literature search and produced an initial draft of the summary table for the literature review. AM checked the entries, edited the table, and produced an initial draft of the main text. Both contributed to the editing of the text.
Funding
Publication of this article was funded in part by the University of Colorado Boulder Libraries.
Acknowledgments
We thank Michael J. Kane for providing thoughtful feedback on this manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Baumeister, R. F., Vohs, K. D., and Tice, D. M. (2007). The strength model of self-control. Curr. Dir. Psychol. Sci. 16, 351–355. doi: 10.1111/j.1467-8721.2007.00534.x
Berkman, E. T. (2020). The last thing you need to know about ego depletion. Psychology Today. Available online at: https://www.psychologytoday.com/us/blog/the-motivated-brain/202012/the-last-thing-you-need-know-about-ego-depletion
Bertrams, A. (2020). A schema-activation approach to failure and success in self-control. Front. Psychol. 11, 2256. doi: 10.3389/fpsyg.2020.02256
Calvillo, D. P., Rodriguez, K., and Nguyen, T. N. (2022). A within-subject test of ego depletion and potential moderators. J. Gen. Psychol. 149, 509–523. doi: 10.1080/00221309.2021.1922341
Carruth, N. P., Ramos, J. A., and Miyake, A. (2023). Does willpower mindset really moderate the ego-depletion effect?: A preregistered replication of Job, Dweck, and Walton (2010). PsyArXiv. doi: 10.31234/osf.io/8cqpk
Carter, E. C., Kofler, L. M., Forster, D. E., and McCullough, M. E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. J. Exp. Psychol. 144, 796–815. doi: 10.1037/xge0000083
Carter, E. C., and McCullough, M. E. (2014). Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated? Front. Psychol. 5, 823. doi: 10.3389/fpsyg.2014.00823
Chow, J. T., Hui, C. M., and Lau, S. (2015). A depleted mind feels inefficacious: Ego-depletion reduces self-efficacy to exert further self-control. Eur. J. Soc. Psychol. 45, 754–768. doi: 10.1002/ejsp.2120
Clarkson, J. J., Otto, A. S., Hirt, E. R., and Egan, P. M. (2016). The malleable efficacy of willpower theories. Pers. Soc. Psychol. Bull. 42, 1490–1504. doi: 10.1177/0146167216664059
Dang, J., Barker, P., Baumert, A., Bentvelzen, M., Berkman, E., Buchholz, N., et al. (2021). A multilab replication of the ego depletion effect. Soc. Psychol. Personal. Sci. 12, 14–24. doi: 10.1177/1948550619887702
Dang, J., Liu, Y., Liu, X., and Mao, L. (2017). The ego could be depleted, providing initial exertion is depleting: A preregistered experiment of the ego-depletion effect. Soc. Psychol. 48, 242–245. doi: 10.1027/1864-9335/a000308
Francis, Z., and Job, V. (2018). Lay theories of willpower. Soc. Personal. Psychol. Compass. 12, e12381. doi: 10.1111/spc3.12381
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., and Inzlicht, M. (2019). Is ego depletion real? an analysis of arguments. Pers. Soc. Psychol. Review. 23, 107–131. doi: 10.1177/1088868318762183
Garrison, K. E., Finley, A. J., and Schmeichel, B. J. (2019). Ego depletion reduces attention control: Evidence from two high-powered preregistered experiments. Pers. Soc. Psychol. Bull. doi: 10.1177/0146167218796473
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., et al. (2016). A multilab preregistered replication of the ego-depletion effect. Perspect. Psychol. Sci. 11, 546–573. doi: 10.1177/1745691616652873
Job, V., Dweck, C. S., and Walton, G. M. (2010). Ego depletion—Is it all in your head? Implicit theories about willpower affect self-regulation. Psychol. Sci. 21, 1686–1693. doi: 10.1177/0956797610384745
Job, V., Walton, G. M., Bernecker, K., and Dweck, C. S. (2013). Beliefs about willpower determine the impact of glucose on self-control. Proc. Natl. Acad. Sci. U.S.A. 110, 14837–14842. doi: 10.1073/pnas.1313475110
Lurquin, J. H., and Miyake, A. (2017). Challenges to ego-depletion research go beyond the replication crisis: a need for tackling the conceptual crisis. Front. Psychol. 8, 568. doi: 10.3389/fpsyg.2017.00568
McClelland, G. H., and Judd, C. M. (1993). Statistical difficulties of detecting interactions and moderator effects. Psychol. Bull. 114, 376–390. doi: 10.1037/0033-2909.114.2.376
Savani, K., and Job, V. (2017). Reverse ego-depletion: acts of self-control can improve subsequent performance in Indian cultural contexts. J. Pers. Soc. Psychol. 113, 589–607. doi: 10.1037/pspi0000099
Schimmack, U. (2016). Replicability report No. 1: Is ego-depletion a replicable effect? [Blog post]. Available online at: https://replicationindex.wordpress.com/2016/04/18/is-replicability-report-ego-depletionreplicability-report-of-165-ego-depletion-articles/
Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366. doi: 10.1177/0956797611417632
Singh, R. K., and Göritz, A. S. (2018). Ego depletion does not interfere with working memory performance. Front. Psychol. 9, 538. doi: 10.3389/fpsyg.2018.00538
Singh, R. K., and Göritz, A. S. (2019). Revisiting ego depletion: moderators and measurement. Basic Appl. Soc. Psych. 41, 1–19. doi: 10.1080/01973533.2018.1530671
Vohs, K. D., Schmeichel, B. J., Lohmann, S., Gronau, Q. F., Finley, A. J., Ainsworth, S. E., et al. (2021). A multisite preregistered paradigmatic test of the ego-depletion effect. Psychol. Sci. 32, 1566–1581. doi: 10.1177/0956797621989733
Vohs, K. D., Baumeister, R. F., and Schmeichel, B. J. (2012). Motivation, personal beliefs, and limited resources all contribute to self-control. J. Exp. Soc. Psychol. 48, 943–947. doi: 10.1016/j.jesp.2012.03.002
Wenzel, M., Lind, M., Rowland, Z., Zahn, D., and Kubiak, T. (2019). The limits of ego depletion: a crossover study on the robustness of performance deterioration in consecutive tasks. Soc. Psychol. 50, 292–304. doi: 10.1027/1864-9335/a000365
Keywords: ego depletion, implicit theories of willpower, willpower beliefs, self-control, replicability
Citation: Miyake A and Carruth NP (2023) On the robustness and replicability of the moderating influence of willpower mindset on the ego-depletion effect: existing evidence is weak at best. Front. Psychol. 14:1208299. doi: 10.3389/fpsyg.2023.1208299
Received: 18 April 2023; Accepted: 24 May 2023;
Published: 15 June 2023.
Edited by:
June J. Pilcher, Clemson University, United StatesReviewed by:
Oulmann Zerhouni, Université de Rouen, FranceCopyright © 2023 Miyake and Carruth. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Akira Miyake, akira.miyake@colorado.edu