From naturalistic neuroscience to modeling radical embodiment with narrative enactive systems

Mainstream cognitive neuroscience has begun to accept the idea of embodied mind, which assumes that the human mind is fundamentally constituted by the dynamical interactions of the brain, body, and the environment. In today’s paradigm of naturalistic neurosciences, subjects are exposed to rich contexts, such as video sequences or entire films, under relatively controlled conditions, against which researchers can interpret changes in neural responses within a time window. However, from the point of view of radical embodied cognitive neuroscience, the increasing complexity alone will not suffice as the explanatory apparatus for dynamical embodiment and situatedness of the mind. We suggest that narrative enactive systems with dynamically adaptive content as stimuli, may serve better to account for the embodied mind engaged with the surrounding world. Among the ensuing challenges for neuroimaging studies is how to interpret brain data against broad temporal contexts of previous experiences that condition the unfolding experience of nowness. We propose means to tackle this issue, as well as ways to limit the exponentially growing combinatoria of narrative paths to a controllable number.

Introduction

The embodied mind view assumes that the phenomenon of human mind is fundamentally constituted by the dynamical interactions of the brain, body, and its environment (Merleau-Ponty, 1962; Varela et al., 1991; Clark, 1997; Chemero, 2009; Barrett, 2011). In this article, we first lay out the path we see leading from conventional to naturalistic neuroscience. Then we discuss what we envision as the next steps on the way toward radically embodied cognitive neuroscience.

Conventional to Naturalistic Neurosciences

Cognitive neurosciences following the conventional experimental paradigms of psychology and psychophysiology have typically departed from simple stimuli that are isolated from bodily and situational contexts. The underlying epistemology is close to that of classical empiricism, with the aim to minimize the number of variables in order to make inductive inferences. This approach is, by definition, unable to handle complex systems of contexts as explanations of observations. Unlike in psychology, where the requirement of ecological validity has been long since recognized (Brunswik, 1955), the limits of computational capabilities have heavily constrained the possibilities of ecologically valid neuroscience. Recently, however, the advanced technological developments have allowed the adoption of a more naturalistic approach. The increasing use of stimuli, whose richness of characteristics and contextual embeddings assumedly corresponds to natural settings, marks a major methodological revision in neuroscience (see e.g., Hari and Kujala, 2009; Zaki and Ochsner, 2009; Maguire, 2012). Instead of experimental designs with isolated stimuli, the paradigm of naturalistic neurosciences is focused on the human cognition as it is engaged in more ecologically valid settings of longer durations, with the purpose to gain access to the brain processes that unfold in time.

So far, the method of free film viewing in the fMRI scanner has allowed observing human brain functions in relation to contexts reminiscent to “real-life” situations (Bartels and Zeki, 2004; Hasson et al., 2004; Malinen et al., 2007). The interpretation of the brain responses to film stimuli is typically based on annotation of the features and content factors of interest in the film at particular time points (Lahnakoski et al., 2012). The advantages of free film viewing are many. Films are, on one hand, highly controllable and can be repeated identically to a large number of people, and, on the other, often tell stories embedded in contexts of everyday experience. The use of naturalistic narrative stimuli in fMRI has already accumulated, for instance, the knowledge on the multiplicity of temporal hierarchies in the brain that may range from milliseconds to tens of seconds (Hasson et al., 2008; Kauppi et al., 2010). These findings allow relating neural behavior in different time scales to temporally extended contextual (or situational) factors, such as narrative comprehension, unfolding in time (Lerner et al., 2011). Further, the naturalistic settings allow studying intersubjectively shared aspects of cognition, such as contextual framing effects on social attributions (Mobbs et al., 2006), or emotional contagion (Nummenmaa et al., 2012), that reveal dependency of experience on contexts and perspectives in social settings.

While movies have turned out to serve the purposes of naturalistic neuroscience to some point, they have major shortcomings with respect to radical embodiment. First, though resembling real-life situations, stimulus movies are typically not interactive, and thus not dynamic in the strict ecological sense. Therefore they cannot fully capture brain responses associated with sensorimotor and social enactment in similar manner as they would be elicited in real-life interaction situations. Secondly, movie stimuli give only limited access to extended temporal contexts of situations against which the cinematic story is experienced at a given transient moment of time. Brain activity is often analyzed through a narrow temporal window capturing not much more than that which is more or less immediately audiovisually present in the narrative content (Figure 1), whereby the analysis runs the risk of missing the subject’s temporally laid out engagement and experiential history conditioning the experience of that which is present.

FIGURE 1

Figure 1. In naturalistic brain experiments with film footage as stimuli, the observed brain activity is typically analyzed against the narrow window of nowness that is accessible through whatever is more or less immediately present, while ignoring the broader temporal contexts, for instance, the experiential history or situatedness of the subject, or the contextual effects of previous events on the comprehension of the immediate present.

Relevant questions related to studying human mind in socially valid contexts include, to what extent does the brain activity of “living-by” other people’s experiences on the screen differ from that of direct interaction between people? As an example from game studies, Kätsyri et al. (2013) have demonstrated the substantial qualitative difference between playing and just watching, motivating the goal of deeper engagement in neuroimaging studies. Currently, methods are being developed for scanning synchronous brain activations of two people who are in direct interaction in magnetoencephalography (MEG) with one another via real-time video (Baess et al., 2012), as well as in direct bodily interaction in a two-person fMRI scanner (Lee et al., 2012). Future will show, how two-person neuroimaging will meet the challenges of making analytical inferences of a system consisting of two “black boxes” instead of one. Meanwhile, in our view, the methodological gap between the one-person movie-viewing to the two-person interaction settings could be bridged with developing dynamically adaptive stimulus systems for managing socially valid interaction settings in more controllable manner. We propose that the explanatory framework of enactment might provide ways for modeling computationally the systemic situatedness of the brain and applying these means as dynamically adaptive stimuli. However, there is no reason to underestimate the methodical and epistemological challenges that will be touched in the following discussion.

Enactive Neuroscience

The notion of enactive cognitive science, as proposed by Varela et al. (1991), refers to the assumption of the mind as a brain-body system inseparably embedded in the world in terms of enactment, i.e., active engagement within that world, and that “cognition is not the representation of a pregiven world by a pregiven mind but is rather the enactment of a world and a mind on the basis of a history of the variety of the actions that a being in the world performs” (p. 9). In their terms, enaction consists of two points: (1) perception as perceptually guided action; and (2) the emergence of cognitive structures “from the recurrent sensorimotor patterns that enable action to be perceptually guided” (Ibid., p. 173). Applied to neuroscience, the radical enactive approach, as it is termed, should imply modeling the reciprocal relationship between embodied cognitive states and the corresponding neuronal activity at the level of large-scale emergent dynamical patterns of activity that cut across the brain, body, and the world (Thompson and Varela, 2001).

In order to fully accommodate enactment in the world, enactive stimuli should not be limited to static representations of the world, but rather to be extended to interactive stimulus models that are adaptive and capable of modeling the reciprocal impact of subject’s enactions to themselves. One possible way is related to the interactive brain hypothesis of Di Paolo and De Jaegher (2012) that suggests self-organizing dynamics of social engagement constituted by states of coordination, and transitions between the states. While using real persons or natural(-like) situations as stimuli may elicit “natural” brain activity, the downside is that the stimulus may remain as another “black box”, this is, as unpredictable and difficult to read and control as the brain. Another is to model the dynamics of relevant socio-cultural exchange (Kaipainen, 1994; Kitayama and Park, 2010), which, however, involves assumptions about cultural dynamics beyond the present focus.

In the following, we discuss the possibility of using narration-based dynamical simulations, such that are able to reflect relevant aspects of the mind’s situatedness and embodiedness in a similar manner as movies do, but which, in addition, are interactive and dynamic while maintaining a certain level of controllability.

Narrative Enactive System

Our proposition to bridge radical embodied cognitive views with naturalistic neuroscience draws from the concept of narrative enactive systems (Tikka, 2008; Kaipainen et al., 2011). It implies the application of dynamically generated or composited naturalistic stimuli for neuroscientific experiments (Tikka et al., 2012) as dynamical models of the live situations, in which the experiencer participates and acts in the framework of a story. Rather than fixed stimulus representation, such as linear movie footage, narrative enactive systems simulate the dynamically changing flow of the environment that is adaptive to the viewer’s bodily experience (perception and action-simulation) of the unfolding narrative. We envision that enactive narrative stimuli could be composited in real-time by a means of, for example, re-combinations of audiovisual media clips from a database (Tikka et al., 2006; Pugliese et al., 2014), algorithmically implemented animations, or game-like avatars with the model described above. This arrangement would constitute an enactive medium, capable of simulating life-like situations, in which the subject is engaged as a participant rather than just an observer.

Unlike traditional neuroscience paradigms, including the one referred to as “naturalistic”, an enactive feedback loop is assumed in the proposed setting. That means taking into account the contexts constituted by experiences of the previous events, implicitly guiding every experience of nowness (Figure 2). We assume that the experience of nowness is related to ongoing processes of memory and attention, corresponding to that which Husserl and later Varela have called retention (Husserl, 1964; Varela, 1999; Figure 2). Following their conceptualization, we take that retention is carried along as backward-looking reflections of the continuous process of enactment.

FIGURE 2

Figure 2. The viewer has a contextualized and situated window of nowness to several levels of narrative, including retentive references to earlier events and eliciting protentive expectations for the coming events. The temporal contexts can be weighted or prioritized in a number of ways, amounting to different interpretations (perspectives) of the narrative. They correspond to different narrative sequences that can be algorithmically composited.

In order to substantiate this idea, imagine thriller film footage that shows a bedside drawer in a hotel room. In this situation, the mere presence of the piece of furniture on the transient screen is not enough to account for the excitement elicited in the viewer that has seen the movie from the beginning. It is essential to include the retentive context of having learned about the gun hidden there in a preceding stage of the narrative. This, in turn, implies the anticipation of a violent event, in Husserlian terms protention, likely to elicit certain kinds of brain responses reflecting alertness and readiness for motor action.

The assumption of enactive media (Kaipainen et al., 2011) implies that the experience itself (involving perceptual and action-related neural networks) steers the narrative via some kind of psychophysiological interface, e.g., eye tracking, electrodermal conductivity, or real-time neurofeedback, reflecting aspects of the holistic experience. At the same time, even the stimulus narrative itself has its own forward-driving inertia, corresponding to how environmental events flow independently of the observer, motivating anticipations (protentions). The methodological advantage is that both the history of such enactions (retention) as well as time-coded annotations of the narrative is known. These can be further aggregated into a time-dependent simulation of the experiential perspective (nowness) at each particular point of stimulus presentation, eventually providing a spectrum of cues for interpreting corresponding brain activity beyond the mere immediate stimulus. A preliminary computational model that simulates such an evolving experiential perspective, termed nowness model, has been introduced by Kauttonen et al. (2014). It consists of (a) decaying memory traces of previous experiences; and (b) a mechanism of refreshing traces via emerging associations. The nowness model, unlike conventional stimuli of neuroimaging studies, would be able to stand for a spectrum of contextualizing narrative factors, such as the hidden gun (referring to the previous example).

A practical challenge is obviously that the combinatoria of narrative paths may well grow too broad to allow sufficient statistical momentum for brain imaging studies, unless constrained to some manageable number. Our preliminary suggestion is to limit the variations to a controllable number of prototypical narrative categories, such a reduction might follow, for example, studies of viewer responses related to different film genres (Grodal, 1997; Visch and Tan, 2009), or player-responses in interactive games (Mathiak and Weber, 2006; Kätsyri et al., 2013).

Discussion

For neurosciences, the implication of the radical embodied cognitive approach is to move toward holistic and systemic study of the brain-world-body dynamics. We propose rethinking the role of stimuli in neuroscience: instead of static representations, should be conceived of as continuously evolving dynamical systems that are capable of connecting the past and future experiences and of accounting for the mutual influences between brain-body and its environment. In our view, Varela’s concept of enactment encapsulates the essence of this relation and allows ways to go beyond today’s naturalistic neuroscience.

Obviously, the proposed approach sets significant challenges for the methods of empirical neurosciences. However, while the complete solution would amount to nothing less than modeling the world around the brain, we believe it possible to isolate partial, focused and controllable dynamic stimulus systems to address essential aspects of radical cognitive neuroscience. This prerequires rich annotations that do not have to be limited to conscious, linguistic, and semantic aspects of the experience (e.g., researcher’s or community’s tags on the stimulus material), but can also include psychophysiological recordings associated with the material (psychophysiological annotations, e.g., heart rate, eye-movements, electrodermal conductivity), environmental conditions (e.g., light, loudness, temperature), and references to cultural contexts (e.g., genre, reviews, background information). Besides rich annotations, the approach requires using the simulation of how previous embodied experiences are present in the experience of newness. Binding together the past and the anticipation of future beyond mere point-like fixed stimuli would allow a temporally situated interpretation of the corresponding observations of brain activity.

Fully conscious of the complexity of translating the proposed approach to actual laboratory work, we are confident that the idea of stimuli that are dynamically adaptive to embodied actions can open new avenues to radical embodied cognitive neuroscience.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This study was made possible by aivoAALTO research project and Aalto Starting Grant of Aalto University, together with Södertörn University and the Foundation for Baltic and East European Studies.

References

Baess, P., Zhdanov, A., Mandel, A., Parkkonen, L., Hirvenkari, L., Mäkelä, J. P., et al. (2012). MEG dual scanning: a procedure to study real-time auditory interaction between two persons. Front. Hum. Neurosci. 6:83. doi: 10.3389/fnhum.2012.00083

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Barrett, L. (2011). Beyond the Brain: How the Body and the Environment Shape Cognition. New Jersey: Princeton University Press.

Google Scholar

Bartels, A., and Zeki, S. (2004). The chronoarchitecture of the human brain—natural viewing conditions reveal a time-based anatomy of the brain. Neuroimage 22, 419–433. doi: 10.1016/j.neuroimage.2004.01.007

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychol. Rev. 62, 193–217. doi: 10.1037/h0047470

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Chemero, A. (2009). Radical Embodied Cognitive Science. Cambridge, MA: MIT Press.

Google Scholar

Clark, A. (1997). Being There: Putting Brain, Body and World Together Again. Cambridge, MA: MIT Press.

Google Scholar

Di Paolo, E. A., and De Jaegher, H. (2012). The interactive brain hypothesis. Front. Hum. Neurosci. 6:163. doi: 10.3389/fnhum.2012.00163

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Grodal, T. K. (1997). Moving Pictures: A New Theory of Film Genres, Feelings and Cognition. New York: Oxford University Press.

Google Scholar

Hari, R., and Kujala, M. V. (2009). Brain basis of human social interaction: from concepts to brain imaging. Physiol. Rev. 89, 453–479. doi: 10.1152/physrev.00041.2007

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., and Malach, R. (2004). Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640. doi: 10.1126/science.1089506

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Hasson, U., Yang, E., Vallines, I., Heeger, D. J., and Rubin, N. (2008). A hierarchy of temporal receptive windows in human cortex. J. Neurosci. 28, 2539–2550. doi: 10.1523/JNEUROSCI.5487-07.2008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Husserl, E. (1964). The Phenomenology of Internal Time Consciousness. [Zur Phänomenologie des Inneren Zeitbewusstseins originates from 1893–1917]. Bloomington: Indiana University Press.

Google Scholar

Kaipainen, M. (1994). Dynamics of Musical Knowledge Ecology. Knowing-What and Knowing-How in the World of Sounds. Helsinki: Acta Musicologica Fennica 19.

Google Scholar

Kaipainen, M., Ravaja, N., Tikka, P., Vuori, R., Pugliese, R., Rapino, M., et al. (2011). Enactive systems and enactive media. Embodied human—machine coupling beyond interfaces. Leonardo J. 44, 433–438. doi: 10.1162/leon_a_00244

CrossRef Full Text | Google Scholar

Kätsyri, J., Hari, R., Ravaja, N., and Nummenmaa, L. (2013). Just watching the game ain’t enough: striatal fMRI reward responses to successes and failures in a video game during active and vicarious playing. Front. Hum. Neurosci. 7:278. doi: 10.3389/fnhum.2013.00278

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kauppi, J.-P., Jääskeläinen, I. P., Sams, M., and Tohka, J. (2010). Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency. Front. Neuroinform. 4:5. doi: 10.3389/fninf.2010.00005

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kauttonen, J., Kaipainen, M., and Tikka, P. (2014). “Model of Narrative Nowness for Neurocinematic Experiments,” in Proceedings 5th Workshop on Computational Models of Narrative (CMN’14). Open Access Series in Informatics (Germany: Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl Publishing), 77–87.

Google Scholar

Kitayama, S., and Park, J. (2010). Cultural neuroscience of the self: understanding the social grounding of the brain. Soc. Cogn. Affect. Neurosci. 5, 111–129. doi: 10.1093/scan/nsq052

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Lahnakoski, J. M., Salmi, J., Jääskeläinen, I. P., Lampinen, J., Glerean, E., Tikka, P., et al. (2012). Stimulus-related independent component and voxel-wise analysis of human brain activity during free viewing of a feature film. PLoS One 7:e35215. doi: 10.1371/journal.pone.0035215

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Lee, R. F., Dai, W., and Jones, J. (2012). Decoupled circular-polarized dual-head volume coil pair for studying two interacting human brains with dyadic fMRI. Magn. Reson. Med. 68, 1087–1096. doi: 10.1002/mrm.23313

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Lerner, Y., Honey, C. J., Silbert, L. J., and Hasson, U. (2011). Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915. doi: 10.1523/jneurosci.3684-10.2011

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Maguire, E. A. (2012). Studying the freely-behaving brain with fMRI. Neuroimage 62, 1170–1176. doi: 10.1016/j.neuroimage.2012.01.009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Malinen, S., Hlushchuk, Y., and Hari, R. (2007). Towards natural stimulation in fMRI—issues of data analysis. Neuroimage 35, 131–139. doi: 10.1016/j.neuroimage.2006.11.015

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Mathiak, K., and Weber, R. (2006). Toward brain correlates of natural behavior: fMRI during violent video games. Hum. Brain Mapp. 27, 948–956. doi: 10.1002/hbm.20234

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Merleau-Ponty, M. (1962). Phenomenology of Perception. Trans. C. Smith. New York: Humanities Press and London: Routledge and Kegan Paul. [Phénoménologie de la perception. Paris: Gallimard, 1945]

Google Scholar

Mobbs, D., Weiskopf, N., Lau, H. C., Featherstone, E., Dolan, R. J., and Frith, C. D. (2006). The kuleshov effect: the influence of contextual framing on emotional attributions. Soc. Cogn. Affect. Neurosci. 1, 95–106. doi: 10.1093/scan/nsl014

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Nummenmaa, L., Glerean, E., Viinikainen, M., Jääskeläinen, I. P., Hari, R., and Sams, M. (2012). Emotions promote social interaction by synchronizing brain activity across individuals. Proc. Natl. Acad. Sci. U S A 109, 9599–9604. doi: 10.1073/pnas.1206095109

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Pugliese, R., Tikka, P., and Kaipainen, M. (2014). “Navigating story ontospace: perspective-relative drive and combinatory montage of cinematic content,” in Expanding Practices in Audiovisual Narrative, eds R. Kelomees and C. Hales (Newcastle: Cambridge Scholars Publishing), 105–127.

Google Scholar

Thompson, E., and Varela, F. J. (2001). Radical embodiment: neural dynamics and consciousness. Trends Cogn. Sci. 5, 418–425. doi: 10.1016/s1364-6613(00)01750-2

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Tikka, P. (2008). Enactive Cinema: Simulatorium Eisensteinense. Helsinki: University of Art and Design Publication Series.

Google Scholar

Tikka, P., Väljamäe, A., de Borst, A. W., Pugliese, R., Ravaja, N., Kaipainen, M., et al. (2012). Enactive cinema paves way for understanding complex real-time social interaction in neuroimaging experiments. Front. Hum. Neurosci. 6:298. doi: 10.3389/fnhum.2012.00298

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Tikka, P., Vuori, R., and Kaipainen, M. (2006). Narrative logic of enactive cinema: obsession. Digit. Creat. J. 17, 205–212. doi: 10.1080/14626260601074078

CrossRef Full Text | Google Scholar

Varela, F. J. (1999). “The specious present: a neurophenomenology of time consciousness,” in Naturalizing Phenomenology: Issues in Contemporary Phenomenology and Cognitive Science, eds J. Petitot, F. J. Varela, B. Pacoud and J.-M. Roy (Stanford, CA: Stanford University Press), 266–329.

Google Scholar

Varela, F. J., Thompson, E., and Rosch, E. (1991). The Embodied Mind. Cambridge, MA: MIT Press.

Google Scholar

Visch, V. T., and Tan, E. S. (2009). Categorizing moving objects into film genres: the effect of animacy attribution, emotional response and the deviation from non-fiction. Cognition 110, 265–272. doi: 10.1016/j.cognition.2008.10.018

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Zaki, J., and Ochsner, K. (2009). The need for a cognitive neuroscience of naturalistic social cognition. Ann. N Y Acad. Sci. 1167, 16–30. doi: 10.1111/j.1749-6632.2009.04601.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar