A commentary has been posted on this article:
Predictive brain signals best predict upcoming and not previous choices
Martin Lages
Stephanie C. Boyle
Katarzyna JaworskaA commentary on
Predicting free choices for abstract intentions
by Soon, C. S., He, A. H., Bode, S., and Haynes, J. D. (2013). Proc. Nat. Acad. Sci. U.S.A. 110, 6217–6222. doi: 10.1073/pnas.1212218110
In a recent study Soon et al. (2013) predicted abstract intentions from fMRI BOLD activities in localized areas of the brain. Activities in a spherical cluster of voxels served as input to a multivariate pattern classifier (linear SVM). The accuracy for predicting the intention to add or subtract two numbers was determined for clusters centered on different voxels. A prediction accuracy of 60% averaged across participants and based on 10-fold cross-validation was achieved for patterns of voxel activities in the medial frontopolar cortex and precuneus up to 4 s before participants reported being consciously aware of their decision. The prediction accuracy in this study was similar to studies on predicting spontaneous left or right motor decisions (Soon et al., 2008; Bode et al., 2011). Since the task demands placed on the participants create similar methodological issues as in previous studies (Lages and Jaworska, 2012), it seems possible that the multivariate classifier picked up sequential information processing between trials (Bode et al., 2012).
Although the average prediction accuracy of 60% returned to chance level for patterns of voxel activity in the two brain areas shortly after the onset of a new trial and remained at 50% between trials, this observation is neither necessary nor sufficient for the absence of sequential information processing. In order to investigate sequential dependencies the outcome of at least one preceding trial and the current trial needs to be taken into account. Depending on task and response, sequential information processing between trials may emerge in distributed form within the default mode network (DMN) at variable time points (Guggisberg and Mottaz, 2013). In the following we illustrate the issue and suggest how the data may be analyzed.
To illustrate sequential effects consider the following inconspicuous sequence of ten responses (S, A, A, A, S, A, S, A, S, S) where A and S stand for freely choosing addition and subtraction, respectively. There are five As and five Ss suggesting a binomial process with rate parameter p = 0.5. However, if we consider the nine subsequent pairs of responses {(S,A), (A,A), …, (S,S)} then we obtain unequal transition probabilities. A trained classifier that predicts the next response from the preceding response would be 3 out of 5 times or 60% correct if the preceding response is A and 3 out of 4 times or 75% correct if the response is S. Starting with a random guess in the first trial, this gives an average prediction accuracy of 65%. In two behavioral studies replicating two different choice tasks (Haynes et al., 2007; Soon et al., 2008) we used subsequent response pairs to train a linear classifier (SVM) and obtained an average prediction accuracy of 61.6 and 64.1%, respectively (Lages and Jaworska, 2012).
When asked to generate a random sequence, people typically alternate between binary responses with a probability of about 0.6 (Lopes, 1982). This response pattern appears to be relevant in Soon et al.'s study (2013) since the only behavioral evidence for memory-less choice in the 17 (out of 34) selected participants is a histogram plotting average frequencies for different lengths of response sequences fitted by an exponential distribution (Figure S1). The authors take the excellent fit as evidence for random performance. Recently Allefeld et al. (2013) released a detailed account of the behavioral data but there are no further details how the data were compiled and fit. Nevertheless, it is discernable from their Figure S1 that the fit represents an exponential function with two parameters rather than an exponential distribution with a single parameter and that observed frequencies do not add up to probability 1.0. In addition, an exponential distribution would only provide a meaningful approximation of the geometrically distributed phase lengths if choosing addition and subtraction were equally probable [p = (1−p) = 0.5]. However, even with a best-fitting rate parameter of 1-exp(−0.826) ≈0.56 the exponential distribution underestimates the relative frequency of alternations (A,S) and (S,A) with phase length 1 as well as repetitions (A,A,S) and (S,S,A) with phase length 2 (see Figure 1). Increased frequencies for short phase lengths are a hallmark of non-random human choice behavior (Wagenaar, 1972; Lopes, 1982; Treisman and Faulkner, 1987; Falk and Konold, 1997) and these characteristics are not only present in the behavioral data of Soon et al. (2013) but also in Soon et al. (2008); Bode et al. (2011), and Haynes et al. (2007) suggesting that free or spontaneous choice tasks result in non-random behavior.
FIGURE 1
Figure 1. Histogram for length of response sequences (phase length or runs) re-plotted as relative frequencies (adapted from Figure S1 in Soon et al., 2013). The data points are fitted by an exponential function with two parameters (red curve) and an exponential distribution with a single parameter (blue curve). The red curve provides an excellent fit (R2 = 0.99) whereas the blue curve underestimates phase length 1 and 2 (R2 = 0.77). See text for explanation.
A related concern arises from the searchlight analyses. If patterns of voxel activities are analyzed within a moving spherical cluster to predict behavioral responses then pre-processing of the data and definition of the searchlight are important (Etzel et al., 2013; Todd et al., 2013). The implementation of regions of interest, temporal constraints (hemodynamic delay), pre-processing (covariates), and data selection can invalidate the results of a searchlight analysis (Kriegeskorte et al., 2009; Lindquist et al., 2009). In Soon et al. (2013) trials were selected (undersampled) in order to balance the mean response rate. It is therefore possible that the searchlight found a cluster of voxels that was predictive of the next response in the context of the preceding response, simply because transitions between successive responses remained unbalanced. A repeated choice task with self-monitoring of the decision process invites sequential dependencies because the observer has to remember goals, constraints, and execution of the task. If, for example, participants shift a decision criterion following each response (Lages and Treisman, 1998, 2010; Lages and Paul, 2006; Treisman and Lages, 2010) or engage in metacognition by recalling the last response before making the next then neural correlates of these response-dependent processes introduce a confound that would be picked up by a searchlight analysis as soon as transitions between response categories are unbalanced.
We recommend that rather than postulating a 50% chance level, prediction accuracy should be tested with a permutation test (Stelzer et al., 2013) and/or separate multivariate classification analyses conditional on the previous response. Only if individual prediction accuracies reliably exceed observable benchmarks such as response bias and transition probabilities can we rest assured that results are not confounded. The interested reader is invited to test predictability of their own free choice behavior by downloading the demo program in the Appendix.
Acknowledgments
Martin Lages was supported by a research grant from The Leverhulme Trust F00-179/BG (UK). Stephanie Boyle and Katarzyna Jaworska were both funded by BBSRC WestBio DTP studentships.
References
Allefeld, C., Soon, C. S., Bogler, C., Heinzle, J., and Haynes, J. D. (2013). Sequential dependencies between trials in free choice tasks. ArXiv:1311.0753v1. Available online at: http://arxiv.org/abs/1311.0753.
Bode, S., He, A. H., Soon, C. S., Trampel, R., Turner, R., and Haynes, J. D. (2011). Tracking the unconscious generation of free decisions using ultra-high field fMRI. PLoS ONE 6:e21612. doi: 10.1371/journal.pone.0021612
Bode, S., Sewell, D. K., Lilburn, S., Forte, J. D., Smith, P. L., and Stahl, J. (2012). Predicting perceptual decision biases from early brain activity. J. Neurosci. 32, 12488–12498. doi: 10.1523/JNEUROSCI.1708-12.2012
Etzel, J. A., Zacks, J. M., and Braver, T. S. (2013). Searchlight analysis: promise, pitfalls, and potential. Neuroimage 78, 261–269. doi: 10.1016/j.neuroimage.2013.03.041
Falk, R., and Konold, C. (1997). Making sense of randomness: implicit encoding as a bias for judgment. Psychol. Rev. 104, 301–318.
Filevich, E., Kühn, S., and Haggard, P. (2013). There is no free won't. Antecedent brain activity predicts decision to inhibit. PLoS ONE 8:e53053. doi: 10.1371/journal.pone.0053053
Guggisberg, A. G., and Mottaz, A. (2013). Timing and awareness of movement decisions: does consciousness really come too late? Front. Hum. Neurosci. 7:1–11. doi: 10.3389/fnhum.2013.00385
Haynes, J. D., Sakai, K., Rees, G., Gilbert, S., Frith, C., and Passingham, R. E. (2007). Reading hidden intentions in the human brain. Curr. Biol. 17, 323–328. doi: 10.1016/j.cub.2006.11.072
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., and Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535–540. doi: 10.1038/nn.2303
Lages, M., and Treisman, M. (2010). A criterion setting theory of discrimination learning that accounts for anisotropies and context effects. Seeing Perceiving 23, 401–434. doi: 10.1163/187847510X541117
Lages, M., and Jaworska, K. (2012). How predictable are “spontaneous decisions” and “hidden intentions?” Comparing classification results based on previous responses with multivariate pattern analysis of fMRI BOLD signals. Front. Psychol. 3:56. doi: 10.3389/fpsyg.2012.00056
Lages, M., and Paul, A. (2006). Visual long-term memory for spatial frequency? Psychon. Bull. Rev. 13, 486–492. doi: 10.3758/BF03193874
Lages, M., and Treisman, M. (1998). Spatial frequency discrimination: visual long-term memory or criterion setting? Vision Res. 38, 557–572. doi: 10.1016/S0042-6989(97)88333-2
Lindquist, M. A., Loh, J. M., Atlas, L. Y., and Wager, T. D. (2009). Modeling the hemodynamic response function in fMRI: efficiency, bias, and mis-modeling. Neuroimage 45, S187–S198. doi: 10.1016/j.neuroimage.2008.10.065
Lopes, L. (1982). Doing the impossible: a note on induction and experience of randomness. J. Exp. Psychol. 8, 626–636. doi: 10.1037//0278-7393.8.6.626
Soon, C. S., Brass, M., Heinze, H. J., and Haynes, J. D. (2008). Unconscious determinants of free decisions in the human brain. Nat. Neurosci. 11, 543–545. doi: 10.1038/nn.2112
Soon, C. S., He, A. H., Bode, S., and Haynes, J.-D. (2013). Predicting free choices for abstract intentions. Proc. Natl. Acad. Sci. U.S.A. 110, 6217–6222. doi: 10.1073/pnas.1212218110
Stelzer, J., Chen, Y., and Turner, R. (2013). Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): random permutations and cluster size control. Neuroimage 65, 69–82. doi: 10.1016/j.neuroimage.2012.09.063
Todd, M. T., Nystrom, L. E., and Cohen, J. D. (2013). Confounds in multivariate pattern analysis: theory and rule representation case study. Neuroimage 77, 157–165. doi: 10.1016/j.neuroimage.2013.03.039
Treisman, M., and Faulkner, A. (1987). Generation of random sequences by human subjects: cognitive operations or psychophysical process. J. Exp. Psychol. Gen. 116, 337–355.
Treisman, M., and Lages, M. (2010). Sensory integration across modalities: how kinaesthesia integrates with vision in visual orientation discrimination. Seeing Perceiving 23, 435–462. doi: 10.1163/187847510X541126
Appendix
After the game is before the game—Josef “Sepp” Herberger (1897-1977, football player and manager)
The demo program can be downloaded from http://www.psy.gla.ac.uk/~martinl/Assets/Software/Predictatron.txt
Please save the text file as Predictatron.m before running it under MatLab (MathWorks Inc., Natick MA). The program illustrates the difficulty for a human decision maker to generate a truly random sequence in a free choice task. Different choice tasks may be simulated by freely choosing between addition/subtraction (Haynes et al., 2007; Soon et al., 2013), left/right (Soon et al., 2008; Bode et al., 2011), or delay/non-delay (Filevich et al., 2013) before pressing the corresponding left or right arrow key on the keyboard. The program records 100 binary responses before it computes the rate of left/right key presses, number of left/right alternations, and left/right repetitions. Then the program determines the prediction accuracy of a linear support vector machine (SVM) using one preceding response to predict the next response (1-back SVM) for 10-fold cross-validation. If the algorithm does not exceed 60% prediction accuracy then two preceding responses are used to predict the next response (2-back SVM). If both prediction accuracies stay below 60% then the participant is considered “unpredictable” and wins against the “Predict-a-tron”. Results of the non-parametric Wald-Wolfowitz runstest() and parametric Lilliefors lillietest() as implemented in MatLab are also reported.
Keywords: free will controversy, sequential dependencies, multivariate pattern analysis, free choice, prediction, searchlight, frontopolar cortex
Citation: Lages M, Boyle SC and Jaworska K (2013) Flipping a coin in your head without monitoring outcomes? Comments on predicting free choices and a demo program. Front. Psychol. 4:925. doi: 10.3389/fpsyg.2013.00925
Received: 21 June 2013; Accepted: 22 November 2013;
Published online: 09 December 2013.
Edited by:
Björn Brembs, University of Regensburg, GermanyReviewed by:
Allen Neuringer, Reed College, USAAlexander Maye, University Medical Center Hamburg-Eppendorf, Germany
Copyright © 2013 Lages, Boyle and Jaworska. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: martin.lages@glasgow.ac.uk