The consistency fallacy and failures of theory embellishment

A commentary on
To disconfirm or not to disconfirm: a null prediction vs. no prediction

by Trafimow, D. (2013). Front. Psychol. 4:733. doi: 10.3389/fpsyg.2013.00733

Trafimow (2013) rejects the notion that data can militate against a theory if the data is wholly irrelevant to the theory. This very reasonable-sounding argument may not hold in the important (and common) case where competing models are being compared. The underlying issues may be clarified by way of example.

Consider the active debate on the contributions of fMRI to cognitive theory (e.g., Mather et al., 2013). It has been argued that fMRI adds little to cognitive theorizing, because data from fMRI are fundamentally irrelevant to cognitive theories pitched at the traditional computational level of analysis (Marr, 1982). A so-called “consistency fallacy” is said to be committed when data is treated as informative simply because some theory is consistent with it (Loosemore and Harley, 2010; Coltheart, 2013). This fallacy putatively arises because no other result could possibly obtain; neuroscientific data is simply irrelevant to cognitive theories at this level of analysis.

Both this argument and Trafimow's (2013) more general variant of it are, however, incorrect when any competing theory assigns a differential prior to the obtained result. Take the general case described by Trafimow (2013): some theory assigns equal likelihood to all possible outcomes of an experiment. (This is just another way of saying the data are irrelevant to it.) The observed data will nonetheless reduce our belief in the original theory as compared to some competing theory, all else being equal, if this competitor had assigned a higher prior to the obtained outcome. Conversely, the observed data will increase our belief in the original theory as compared to a competing theory, if this competitor had assigned a lower prior to the obtained outcome. This argument holds regardless of whether the observed outcome is a significant difference or a null effect; Bayes' rule of course applies in both these conditions.

That said, it is true that no evidence can be offered regarding existing theories if they all assign equal likelihood to all outcomes, or if they cannot be subject to model comparison. In practice, such a situation might less often reflect sloppy inference than a failure to adequately articulate, develop or embellish a theory.

Acknowledgments

Thanks to David Badre for helpful discussion, and the National Institute of Neurological Disease and Stroke (R01 NS065046) for funding.

References

Coltheart, M. (2013). How can functional neuroimaging inform cognitive theories? Perspect. Psychol. Sci. 8, 98–103. doi: 10.1177/1745691612469208

CrossRef Full Text

Loosemore, R., and Harley, T. (2010). “17 brains and minds: on the usefulness of localization data to cognitive psychology,” in Foundational Issues in Human Brain Mapping, eds S. J. Hanson and M. Bunzl (Cambridge, MA: MIT Press), 217–240. doi: 10.7551/mitpress/9780262014021.003.0018

CrossRef Full Text

Marr, D. (1982). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. New York, NY: Henry Holt and Co, Inc.

Mather, M., Cacioppo, J. T., and Kanwisher, N. (2013). 20 years of fMRI—what has it done for understanding cognition? [Special Section]. Perspect. Psychol. Sci. 8, 41–113. doi: 10.1177/1745691612469036

CrossRef Full Text

Trafimow, D. (2013). To disconfirm or not to disconfirm: a null prediction vs. no prediction. Front. Psychol. 4:733. doi: 10.3389/fpsyg.2013.00733

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text