This article is a commentary on:
Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?
Matti T. J. Heino
Eiko I. Fried
Etienne P. LeBel- 1Department of Social Sciences, University of Tampere, Tampere, Finland
- 2Department of Social Research, University of Helsinki, Helsinki, Finland
- 3Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
- 4Department of Psychology, Western University, London, Canada
A commentary on
Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?
by Iso-Ahola, S. E. (2017). Front. Psychol. 8:879. doi: 10.3389/fpsyg.2017.00879
[F]alsifiability and replication are of secondary importance to advancement of scientific fields.
A cultural divide is forming in psychology: while some researchers are skeptical of research practices and standards prior to the current replicability crisis (e.g., Munafò et al., 2017), others have strong faith in the published literature (e.g., Gilbert et al., 2016). This difference of perspectives drives many current debates, and is also visible in a new paper in Frontiers in Psychology by Iso-Ahola, titled “Reproducibility in psychological science: When do psychological phenomena exist”, in which the author states that prior findings cannot be all dismissed as flukes “because they were published in the best journals of social psychology” (Iso-Ahola, 2017).
In the piece, Iso-Ahola sets out the idea that experiments are only able to show evidence for phenomena, but can never “prove a negative.” This makes it impossible to falsify psychological ideas like the ego depletion effect (Baumeister et al., 1998). He argues that “reproducibility in psychology is unattainable,” and “that psychological phenomena, by their nature, are not fully reproducible” because humans “can be astonishingly simple or irreducibly complex at various times.” Iso-Ahola further claims that researchers have largely focused on isolated indicators—such as effect size and replicability—to determine whether psychological phenomena exist. Contrasting these practices, he proposes 10 criteria to vet psychological phenomena. Instead of attempting to perform exact replications that make little sense since they cannot disprove theories, the author argues that researchers should use conceptual replications to “attempt to establish a phenomenon's boundary conditions”. Iso-Ahola's default position seems to be that all phenomena exist. He puts the burden of proof on the skeptics, and not on those who claim they have identified psychological phenomena, arguing that “nobody has provided a theoretically and logically rigorous rationale and justification why ego depletion as a phenomenon should and would not exist.”
We agree that some of the language around replication could be more precise and that direct replications could be improved in several respects (Klein et al., 2014; LeBel et al., 2017). We disagree, however, with Iso-Ahola's positions that falsifiability is “of secondary importance” to scientific progress, that psychological phenomena are in principle not “fully reproducible[sic]”, or that one should conduct conceptual replications before ascertaining that there really is something to replicate (and the concepts are sufficiently clear; see Lurquin and Miyake, 2017). Falsification is what makes science self-correcting (Popper, 2005; LeBel et al., in press), and replication is possible if researchers clearly specify which conditions are crucial for their hypothesis to hold. Risky tests, for example via sufficiently methodologically similar direct replications, are necessary to produce the insight that some of our ideas may well be wrong; in the absence of such rigorous replications, psychology turns into astrology. Such replications are especially important given that modern psychology has seen numerous highly influential findings not replicate when tested via high-powered and transparently executed direct replications (LeBel et al., in press see curated list of unsuccessful replications at https://osf.io/8srcd/). Iso-Ahola, however, maintains that “phenomena's existence should not be defined by any index of reproducibility of findings.” In our opinion, that is an unnecessary lowering of scientific standards. Instead, falsification via replicability tests forces the investigators to consider more fitting measurement or modeling approaches (person-level time series designs, e.g., Molenaar and Campbell, 2009; highly-repeated within-person designs, e.g., Whitsett and Shoda, 2014), better experimental designs, and more sophisticated paradigms.
Now, we should not pretend falsification is easy, especially in psychology. It is challenging to confirm that auxiliary hypotheses—e.g., soundness of measurement and experimental procedures—are reasonable (Meehl, 1990; LeBel and Peters, 2011; Earp and Trafimow, 2015). This is naturally a challenge for original studies, too. On the bright side, approaches to executing direct replications are becoming more sophisticated. For example, CurateScience.org (LeBel et al., 2017) now presents “active sample evidence” including positive controls, manipulation checks, and measurement integrity (e.g., internal consistency estimates) as a way to help confirm sound auxiliaries (for an example of a replication reporting a positive control, see Sanchez et al., 2017). In addition, links to open materials/data are provided to verify the integrity of replications.
Falsification is also difficult because replications may fail due to problems with operationalizations of the focal constructs. Another reason for lack of replicability and generalizability of psychological phenomena that has been largely overlooked in the extant literature is their complexity: psychological processes such as emotions, cognitions, personality characteristics, or mental disorders are highly multi-causal, with thousands of determinants that often have very small effects on behavior. It seems like Iso-Ahola refers to this, when he talks about the “subtle, elusive” nature of psychological phenomena. But the study of complex dynamic systems (Bar-Yam, 1997; Vallacher and Nowak, 2008) has been widely successful in other areas of research in the last decade (e.g., ecology, biology, and physics), and offers a way forward for psychology too. Complex systems can be characterized as webs of interdependent self-organizing parts whose interactions give rise to emergent properties (Borsboom and Cramer, 2013). Some variations of this perspective have already been brought to at least psychopathology (Fried et al., 2017), personality (Mõttus and Allerhand, 2017), intelligence (Van Der Maas et al., 2017), development (Smith and Thelen, 2003), language (Beckner et al., 2009), and public health (Resnicow and Page, 2008).
The consequence of complexity should not be to abandon replicability, but to embrace it—via sophisticated theories, improved study designs and operationalizations, and modern statistical models that adequately account for the heterogeneity and complexity of psychological phenomena. Replicability is the only way to self-correct our understanding of psychological phenomena in a productive and cumulative fashion.
Author Contribution
All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding
MH is funded by Academy of Finland Grant no. 295765. EF is funded by the European Research Council Consolidator Grant no. 647209.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Baumeister, R. F., Bratslavsky, E., Muraven, M., and Tice, D. M. (1998). Ego depletion: is the active self a limited resource? J. Pers. Soc. Psychol. 74:1252.
Beckner, C., Blythe, R., Bybee, J., Christiansen, M. H., Croft, W., Ellis, N. C., et al. (2009). Language is a complex adaptive system. Lang. Learn. 59, 1–26. doi: 10.1111/j.1467-9922.2009.00533.x
Borsboom, D., and Cramer, A. O. J. (2013). Network analysis: an integrative approach to the structure of psychopathology. Annu. Rev. Clin. Psychol. 9, 91–121. doi: 10.1146/annurev-clinpsy-050212-185608
Earp, B. D., and Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Quant. Psychol. Measure. 6:621. doi: 10.3389/fpsyg.2015.00621
Fried, E. I., van Borkulo, C. D., Cramer, A. O., Boschloo, L., Schoevers, R. A., and Borsboom, D. (2017). Mental disorders as networks of problems: a review of recent insights. Soc. Psychiatry Psychiatr. Epidemiol. 52, 1–10. doi: 10.1007/s00127-016-1319-z
Gilbert, D. T., King, G., Pettigrew, S., and Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science 351, 1037–1037. doi: 10.1126/science.aad7243
Iso-Ahola, S. E. (2017). Reproducibility in psychological science: when do psychological phenomena exist? Front. Psychol. 8:879. doi: 10.3389/fpsyg.2017.00879
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., et al. (2014). Theory building through replication: response to commentaries on the “Many Labs” replication project. Soc. Psychol. 45, 307–310.
LeBel, E. P., Berger, D., Campbell, L., and Loving, T. J. (in press). Falsifiability is not optional. J. Pers. Soc. Psychol. Available online at: https://osf.io/preprints/psyarxiv/dv94b/
LeBel, E. P., and Peters, K. R. (2011). Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice. Rev. Gen. Psychol. 15, 371–379. doi: 10.1037/a0025172
LeBel, E. P., Vanpaemel, W., McCarthy, R., Earp, B. D., and Elson, M. (2017). A Unified Framework for Quantifying the Reproducibility, Robustness, Replicability, and Generalizability of Published Scientific Findings. Available online at: http://curatescience.org/docs/CurateScience-NewFramework[3.0.1].pdf
Lurquin, J. H., and Miyake, A. (2017). Challenges to ego-depletion research go beyond the replication crisis: a need for tackling the conceptual crisis. Front. Psychol. 8:568. doi: 10.3389/fpsyg.2017.00568
Meehl, P. E. (1990). Appraising and amending theories: the strategy of Lakatosian defense and two principles that warrant it. Psychol. Inq. 1, 108–141.
Molenaar, P. C., and Campbell, C. G. (2009). The new person-specific paradigm in psychology. Curr. Dir. Psychol. Sci. 18, 112–117.
Mõttus, R., and Allerhand, M. (2017). SAGE Handbook of Personality and Individual Differences, Vol. 1, The Science of Personality and Individual Differences. London: SAGE.
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P., et al. (2017). A manifesto for reproducible science. Nat. Hum. Behav. 1:0021. doi: 10.1038/s41562-016-0021
Resnicow, K., and Page, S. E. (2008). Embracing chaos and complexity: a quantum change for public health. Am. J. Public Health 98, 1382–1389. doi: 10.2105/AJPH.2007.129460
Sanchez, C., Sundermeier, B., Gray, K., and Calin-Jageman, R. J. (2017). Direct replication of Gervais and Norenzayan (2012): no evidence that analytic thinking decreases religious belief. PLoS ONE 12:e0172636. doi: 10.1371/journal.pone.0172636
Smith, L. B., and Thelen, E. (2003). Development as a dynamic system. Trends Cogn. Sci. 7, 343–348. doi: 10.1016/S1364-6613(03)00156-6
Vallacher, R. R., and Nowak, A. (2008). “The dynamics of human experience: fundamentals of dynamical social psychology,” in Chaos and Complexity in Psychology, eds S. J. Guastello, M. Koopmans, and D. Pincus (Cambridge: Cambridge University Press), 370–401.
Van Der Maas, H., Kan, K.-J., Marsman, M., and Stevenson, C. E. (2017). Network Models for Cognitive Development and Intelligence. doi: 10.20944/preprints201701.0107.v1
Keywords: replicability crisis, replication, complexity, complex systems, falsification
Citation: Heino MTJ, Fried EI and LeBel EP (2017) Commentary: Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? Front. Psychol. 8:1004. doi: 10.3389/fpsyg.2017.01004
Received: 09 May 2017; Accepted: 31 May 2017;
Published: 22 June 2017.
Edited by:
Martin S. Hagger, Curtin University, AustraliaReviewed by:
David Trafimow, New Mexico State University, United StatesCopyright © 2017 Heino, Fried and LeBel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Matti T. J. Heino, matti.tj.heino@gmail.com