Pia Tikka1* Aleksander Väljamäe2,3 Aline W. de Borst1 Roberto Pugliese4 Niklas Ravaja5,6 Mauri Kaipainen7 Tapio Takala4- 1NeuroCine, Department of Film, Television and Scenography, School of ARTS, Aalto University, Helsinki, Finland
- 2Neuropsychology Laboratory, University of Graz, Graz, Austria
- 3NeuroGenetics Media Lab, CIANT, Prague, Czech Republic
- 4Department of Media Technology, School of Science, Aalto University, Espoo, Finland
- 5Department of Social Research and Helsinki Institute for Information Technology, University of Helsinki, Helsinki, Finland
- 6School of Business, Aalto University, Helsinki, Finland
- 7School of Communication, Media and Information Technology, Södertörn University, Huddinge, Sweden
We outline general theoretical and practical implications of what we promote as enactive cinema for the neuroscientific study of online socio-emotional interaction. In a real-time functional magnetic resonance imaging (rt-fMRI) setting, participants are immersed in cinematic experiences that simulate social situations. While viewing, their physiological reactions—including brain responses—are tracked, representing implicit and unconscious experiences of the on-going social situations. These reactions, in turn, are analyzed in real-time and fed back to modify the cinematic sequences they are viewing while being scanned. Due to the engaging cinematic content, the proposed setting focuses on living-by in terms of shared psycho-physiological epiphenomena of experience rather than active coping in terms of goal-oriented motor actions. It constitutes a means to parametrically modify stimuli that depict social situations and their broader environmental contexts. As an alternative to studying the variation of brain responses as a function of a priori fixed stimuli, this method can be applied to survey the range of stimuli that evoke similar responses across participants at particular brain regions of interest.
Introduction
“David observes Sally entering a coffee room filled with lively conversation of colleagues. Immediately the conversation ends and everybody leaves hastily. Sally is left alone. David sees how she fights the tears while sipping her coffee, and cannot help feeling sorry for her. The scene repeats every day. Then once again, the coffee room is crowded. David is already expecting Sally's entrance. Suddenly someone looks directly at David, as if realizing his presence. This time, as Sally enters and everybody else leaves, the person who has shared a look with David hesitates. Glances at David but then leaves together with others, which makes David feel disappointed. Next time during the coffee break, the person with whom David earlier shared a look, unexpectedly remains and greets Sally. The gratefulness on Sally's face, and a quick glance that the fellow worker shares with David, makes David feel that his engagement with the events has made a difference.”
Consider that David, instead of being physically in the same space as Sally and despite of lying motionless in an fMRI scanner, is nevertheless enactively engaged in the setting. That is, he not only reacts to but also influences the emotional behavior of the screen characters. According to our vision, this setting addresses socially synchronized and shared everyday experiences online by means of engagement characteristic for cinema viewing.
Neuroscience of Social Interaction
In experimental settings social behavior is typically stripped out of its situational and embodied contexts due to methodological reasons, thus risking their ecological validity (Neisser, 1976; Gibson, 1979). Recent paradigms of social neuroscience experiments try to overcome these problems by, for example, letting participants in a functional magnetic resonance imaging (fMRI) scanner interact with someone inside another scanner (Montague et al., 2002), or outside of the scanner (Redcay et al., 2010). Also magnetoencephalographic (MEG-to-MEG) real-time interaction recordings have been successfully conducted (Baess et al., 2012). Further, the development of two-person fMRI head-coils allows a major step forward in introducing physical presence and intimacy to online interaction settings in fMRI (Lee et al., 2010, 2011). But adding another person into the scanner still leaves these settings far from capturing context-dependency of social behavior. Since the biological brain continuously simulates the phenomena of surrounding world (Rizzolatti and Arbib, 1998; Iacoboni et al., 2005; Rizzolatti and Fabbri-Destro, 2008; Cattaneo and Rizzolatti, 2009; Hari and Kujala, 2009), brain activity cannot be fully understood without a model that addresses even the dynamics of the social environment.
Neurocinematics
Cinema serves the interest of social neuroscience because it simulates life by its very nature (Tan, 1996; Mar and Oatley, 2008; Tikka, 2008; Grodal, 2009). Neurocinematics, so termed by Hasson et al. (2008b) refers to neuroimaging experiments that use cinematic stimuli to study human behavior. So far, neurocinematic experiments have shown that different viewers' attention to socially determining aspects of the story, often communicated by facial expressions and bodily gestures, is highly correlated (Bartels and Zeki, 2003, 2004; Hasson et al., 2004, 2008a,b, 2010; Bartels et al., 2008; Jääskeläinen et al., 2008; Lahnakoski et al., 2012).
Regarding cinema-viewing in fMRI, we consider two particular conditions essential to social validity. The method of unconstrained free viewing in fMRI allows viewer's emotional immersion into the cinematic world without disturbing breaks. This condition is not different from the reported experiences of presence in virtual reality experiments (Hoffman et al., 2003; Sanchez-Vives and Slater, 2005; Baumgartner et al., 2008). The viewer's engagement with the fictional characters may be understood as parasocial interaction in the sense of Horton and Wohl (1956), or as a kind of inter-reality, an experience that converges real and fictional worlds (Riva et al., 2010). As another aspect of social validity we take the characteristic of immediate two-way responsiveness. This is difficult to achieve with traditional cinema material with a fixed storyline. While the paradigm of interactive cinema has introduced various means of combinatory and generative narrative solutions to this issue, it has another problem: the conscious decision-making expected from the viewers often interrupts the narrative flow. The dilemma we suggest to solve with the enactive cinema setting is how to model participants' engagement in the social situations created by cinematic scenes without destroying the emotional immersion.
Enactive Cinema
While regular cinema may be regarded as simulation of life situations, enactive cinema takes the simulation one step further by letting the viewer's experience influence the narrative in real-time. Being enactive refers to engagement that is more holistic than being interactive. The origin of the notion goes back to the idea of embodied interaction between a subject and another entity as discussed by Varela et al. (1991). In an enactive system, changes in the psychophysiological reactions of participants (enactors) that are assumed to represent implicit and unconscious reactions of the mind, determine the changes made to the narrative presentation in real-time (Kaipainen et al., 2011). The related concept of “enactive cinema” was first introduced in the context of cinema installation “Obsession” (Tikka, 2005). This installation relied on (1) the tracking of spectator's real-time physiological responses, such as heart rate and electrodermal activity, which controlled (2) the algorithm-driven montage of the cinematic expression, which again influenced the spectator's experience, and so on (Tikka et al., 2006). The continuous follow-up of the changes in the participants' bodily responses drives algorithmic modification of the cinematic narration in such a manner that the viewpoints of the ongoing events may change, for example, from suspense to romantic expectations. As a distinction to what is typically called interactive media, in enactive media the participants are not directly in control of the narrative. Enactive cinema can be regarded as modeling socio-emotional dialogs by means of modifying situational factors as a function of participants' experience. The advantage of enactive cinema over conventional linear narration for real-time social neuroscience experiments lies in the systemic interdependence due to two-way feedback, which addresses the socio-emotionally engaging nature of cinema as a simulation of the situatedness in the “real” world.
Real-Time Physiological Feedback
Part of the methodology that can be used to implement an enactive cinema system stems from the vast body of experimental and clinical research using physiology-based real-time feedback since the 1970's (e.g., Nowlis and Kamiya, 1970; Manuck et al., 1975). By presenting participants with feedback of their own physiological state (heart rate, breathing, brain activity, etc.) control over these states can be learnt (e.g., deCharms et al., 2004; Weiskopf et al., 2004). Recently, real-time functional magnetic resonance imaging (rt-fMRI; Cox et al., 1995) has allowed neural activity of multiple brain regions to be used as feedback (see Weiskopf, 2011, for overview). Due to the development of real-time connectivity analysis (Lee et al., 2012) and multivariate classification (Hollmann et al., 2011) real-time fMRI can go beyond mere manipulation of region-of-interest based activity by allowing network analyses and actual prediction of behavior. These neurofeedback methods seem particularly suitable for the investigation of emotions and social behavior (Posse et al., 2003; Johnston et al., 2010; Sarkheil et al., 2011; Veit et al., 2011).
The measures of neural activity can be complemented by peripheral physiological and behavioral signals that give access to emotional and attentional states: e.g., heart-rate and electrodermal activity may indicate emotional arousal, facial electromyography (EMG) activity can index affect, and eye-tracking inform about the direction of overt attention (see e.g., Davidson, 2004; Ravaja, 2004; Heller et al., 2011). Real-time feedback has been explored in various virtual reality settings. For example, eye-tracking has been used to reflect participants' real-time response to the behavior of a virtual character in fMRI (Wilms et al., 2010). Outside of scanner, the motion-capture experiment of Pugliese and Lehtonen (2011) explored a feedback loop of action and reaction between a human participant and a virtual character by means of imitation and designed rules. In the video game Ghost in the Cave the player can control the emotional state of a virtual character (avatar) by using nonverbal acoustical or motion cues (Rinman et al., 2004). Moreover, emotionally adaptive games (e.g., Emoshooter) dynamically change their design depending on the player's emotional state (Kuikkaniemi et al., 2010).
However, the proposed enactive approach goes beyond the previously described paradigms by combining physiological feedback techniques with socially engaging cinematic stimuli. While sharing many techniques with the rapidly developing area of Brain Computer Interfaces (BCI) for neuroscience of social interaction (Mattout, 2012) as well as with unconscious man-machine control loops (e.g., Kaplan et al., 2005), the enactive cinema emphasizes more holistic human-system dialogs.
Enactive Cinema for Social Real-Time Neuroscience
Rather than studying the variation of brain responses as a function of a-priori fixed stimuli we suggest the study of brain activity as a function of dynamically adapted stimuli. Rt-fMRI feedback reflecting the viewer's idiosyncratic brain activity patterns, potentially combined with their peripheral physiological responses, can be applied to modify cinematic stimuli that address ranges of socially engaging situations. The comparison of parametric variations in cinematic stimuli with the synchronous brain data will augment the understanding of the dynamical context-dependency of the social mind.
Enactive Cinema System
The enactive cinema system includes (1) real-time acquisition and (2) analysis of physiological response data, which is further fed into an (3) “interpretation toolbox” that modifies the stimulus stream so as to trigger desired responses based on previous knowledge of brain behavior (see Figure 1).
Figure 1. Diagram of the enactive setting: (A) the viewer's physiological responses are analyzed in real-time, (B) an “interpretation toolbox” compares them with previously known behaviors and selects cinematic elements that are expected to lead toward a desired response, (C) real-time montage produces continuous cinematic stimulus projected back to the participant, whose brain responses are scanned (A), and so on.
Enactive rt-fMRI Feedback
The availability of real-time analysis methods is constantly increasing, thereby greatly enhancing the possibilities of an enactive cinema system. Next to the established real-time analyses of ROIs, several other properties of fMRI data can be exploited, such as functional connectivity (e.g., rt-ICA; Ma et al., 2012), effective connectivity (Lee et al., 2012), pattern classification (Hollmann et al., 2011; LaConte, 2011), as well as a combination of brain data and peripheral physiology (Wilms et al., 2010; Voyvodic et al., 2011). This entails that feedback to the system can be more than a one dimensional increase-decrease relationship of a single region, allowing more fine-grained reflection of the mental state. For example, using real-time ICA the involvement of specific functional networks can be defined. While a single region can be involved in several different cognitive functions, the identification of a functional network will give the system more information about the ongoing cognitive processes, allowing for more detailed feedback. The accuracy of the identification of the cognitive, emotional, and motivational states of an individual can be enhanced by combining several physiological indices with the rt-fMRI data, as the advantages of response patterns (or profiles) over single physiological measures are well established (e.g., Ravaja, 2004). For example, employing several physiological signals, neural networks (e.g., probabilistic neural network) have successfully been trained to recognize emotions when viewing films (Arroyo-Palacios and Romano, 2010). As an example of potential settings, assuming that each participant's brain activation correlates to some extent with a particular type of experience (e.g., fear; Ehrsson et al., 2007), one may optimize the similarity across different participant's brain activations by means of stimuli personalized in real time.
Enactive Cinematic Stimuli
We envision two technical solutions for the real-time presentation of enactive cinematic content in fMRI: (1) the fine-grained recombination of existing cinematic elements, and (2) algorithmic generation of cinematographic characters and images by adapting methods from the fields of animation, gaming and virtual reality research (e.g., Parsons and Courtney, 2011).
The recombination of cinematic elements in real-time allow varying the contextualizing conditions of social interaction within the experimental focus. Its prerequisite is the rich time-coded metadata associated with the content, which enables the retrieval of video footage from a content database and the sorting of this footage into a coherent sequence. The rough annotation of the content with attributes (or “tags”) such as low level perceptual features (e.g., visual, auditory), ecologically valid features (e.g., hands, faces, basic emotions) and higher level features (e.g., social, psychological, cultural) may not suffice to the generation of fine-grained coherent narrative sequences that meet the criteria of social validity.
Based on the ontospace approach (Kaipainen et al., 2008; Kaipainen and Hautamäki, 2011), a metadata approach to narrative has been elaborated (Pugliese et al., 2012), where meaning-bearing annotations of content are described by means of graded values, referring to the prominence of particular content-related features. These values may correspond to above mentioned annotations or, for example, rules of cinematic storytelling (e.g., flashbacks, emotion cues, or temporal ellipses). They are conceived of as narrative dimensions that altogether define a space of narrative metadata, allowing each narrative element to be described in terms of its narrative coordinates (Pugliese et al., 2012). The advantage of this approach is that it allows a dynamical stimulus montage as an element-to-element trajectory through the narrative medatata space, which in turn is shaped by a parametrically controllable perspective, a set of weights associated with one for each dimension. In this space elements similar to each other with respect to the weighted dimensions lie in each other's proximity. Insofar the metadata describes the socially valid aspects of the narration, the approach guarantees a stimulus montage that is not only continuous but even socially coherent. Enactive cinema installation Obsession (Tikka, 2005) can be regarded as a preliminary proof of concept.
The described method can also be applied to generate desired behavior of dynamically controlled stimulus characters, or avatars that can be used to model a partner in a social dialogue. In this case, content-describing narrative dimensions correspond to generative parameters of the avatar. This approach relies on findings of recent fMRI experiments that have revealed the sensitivity of brain functions associated with viewing artificial human-like agents in comparison to humans (Moser et al., 2007; Chaminade et al., 2010; Cheetham et al., 2011; Saygin et al., 2011), confirming the observation of the “uncanny valley” by Mori (1970). However, as the recent technological developments render virtual avatars more human-like than ever before (Rizzo et al., 2004; Alexander et al., 2010), we envision that the technology can eventually bridge the uncanny valley.
Toward Implementation of Enactive Cinema in fMRI
How enactive cinema may be harnessed to study complex social settings in fMRI can be exemplified on the basis of Singer et al. (2006), who showed that after perceiving two people being fair or unfair in an economical game and then seeing both persons receiving painful electrical shocks, two types of response patterns emerged in the participants. The female subjects showed empathy-related brain activity to seeing both fair and unfair persons receiving pain, while the males showed reward-related activity to seeing an unfair person receiving pain (revenge). Applying the enactive cinema setting to this phenomenon, it is possible to study the context-dependency of empathy and revenge. Based on the real-time emerging response patterns in e.g. anterior insula (empathy) or orbito-frontal cortex (revenge) the stimuli can be adapted online for each individual. For example, when participants show a low blood oxygen level dependent (BOLD) signal in anterior insula the system computes, using algorithms and metadata (prior knowledge), dynamical modifications of the cinematic stimulus (e.g., adding personal background information on the unfair person) in order to increase activity/empathy. By incrementally adapting the stimulus, the desired response can be obtained in both males and females. Empathy can then be studied by looking at similarities and differences between the stimuli (e.g., additional information, emotion cues, context, etc.) that made brain responses of the males similar to those of the females.
Similarly, in the case described in the introduction, the fMRI participant's physiological epiphenomena would be associated with David's empathy toward Sally when seeing her socio-emotionally painful situation. Changes in David's biometric measures of arousal as well as neural activity of his limbic regions, perhaps even in some of the brain networks that relate to moral judgments (Parkinson et al., 2011), could be interpreted as indications of changes in his attitude toward the depicted scene. Depending on the goals of the experiment, the stimulus stream could be modified by priming David's perspective to Sally for example by adding explanatory background information concerning Sally's previous activities, or by manipulating the degree of mental violence of the scene, or adjusting the duration of the painful situation.
Due to the two-directionality of the enactive system, the online adjustment of the cinematic stimuli described above on the one hand, and the real-time manipulation of neural activations on the other hand, allow a novel kind of analysis of the brain's functional activity and connectivity versus parametric changes in the stimulus stream.
Conclusion
We have proposed an approach to neuroscience of social interaction that focuses on shared social engagement in dynamical contexts. It is based on the potential of cinema to model everyday situations of life. In order to apply cinematic stimuli in fMRI settings we have suggested the method of generation of cinematic stimulus stream in real-time by means of re-combining existing elements algorithmically, with potential supplementary elements such as animations or avatars. The stream is controlled by the participants' enactivity, i.e., their psychophysiological epiphenomena of experience. For the neuroscience of social interactions enactive setup offers for the first time a means to generate contextualized, socially valid stimulus material under dynamical control, allowing observation of brain activity in the course of a social dialog. Although we acknowledge many technological challenges, however, due to the rapidly evolving real-time implementations both in the field of neuroimaging as well as that of computer generated audiovisual media, we envision that the implementation of an enactive system is only a matter of time.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
Supported by research project aivoAALTO, Aalto University; NeuroCine Aalto Starting Grant; project Enactive Media, the Academy of Finland (contract no. 128132); Södertörn University. Second author has been supported by EC NoE grant GALA (FP7-ICT-2009-5-258169). We thank for discussions Riitta Hari and Raimo Sepponen, Aalto University, and for assistance with the figure we owe thanks to Mika Seppä, Aalto University.
References
Alexander, O., Rogers, M., Lambeth, W., Chiang, J. Y., Ma, W. C., Wang, C. C., et al. (2010). The digital Emily project: achieving a photorealistic digital actor. IEEE Comput. Graph. Appl. 30, 20–31.
Arroyo-Palacios, J., and Romano, D. M. (2010). Bio-affective computer interface for game interaction. IJGCMS 2, 16–32.
Baess, P., Zhdanov, A., Mandel, A., Parkkonen, L., Hirvenkari, L., Mäkelä, J. P., et al. (2012). MEG dual scanning: a procedure to study real-time auditory interaction between two persons. Front. Hum. Neurosci. 6:83. doi: 10.3389/fnhum.2012.00083
Bartels, A., and Zeki, S. (2003). Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75–85.
Bartels, A., and Zeki, S. (2004). The chronoarchitecture of the human brain–natural viewing conditions reveal a time-based anatomy of the brain. Neuroimage 22, 419–433.
Bartels, A., Zeki, S., and Logothetis, N. K. (2008). Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain. Cereb. Cortex 18, 705–717.
Baumgartner, T., Speck, D., Wettstein, D., Masnari, O., Beeli, G., and Jäncke, L. (2008). Feeling present in arousing virtual reality worlds: prefrontal brain regions differentially orchestrate presence experience in adults and children. Front. Hum. Neurosci. 2:8. doi: 10.3389/neuro.09.008.2008
Chaminade, T., Zecca, M., Blakemore, S. J., Takanishi, A., Frith, C. D., Micera, S., et al. (2010). Brain response to a humanoid robot in areas implicated in the perception of human emotional gestures. PLoS ONE 5:e11577. doi: 10.1371/journal.pone.0011577
Cheetham, M., Suter, P., and Jäncke, L. (2011). The human likeness dimension of the “uncanny valley hypothesis”: behavioral and functional MRI findings. Front. Hum. Neurosci. 5:126. doi: 10.3389/fnhum.2011.00126
Cox, R. W., Jesmanowicz, A., and Hyde, J. S. (1995). Real-time functional magnetic resonance imaging. Magn. Reson. Med. 33, 230–236.
Davidson, R. J. (2004). What does the prefrontal cortex “do” in affect: perspectives on frontal EEG asymmetry research. Biol. Psychol. 67, 219–233.
deCharms, R. C., Christoff, K., Glover, G. H., Pauly, J. M., Whitfield, S., and Gabrieli, J. D. (2004). Learned regulation of spatially localized brain activation using real-time fMRI. Neuroimage 21, 436–443.
Ehrsson, H. H., Wiech, K., Weiskopf, N., Dolan, R. J., and Passingham, R. E. (2007). Threatening a rubber hand that you feel is yours elicits a cortical anxiety response. Proc. Natl. Acad. Sci. U.S.A. 104, 9828–9833.
Gibson, J. J. (1979). The Ecological Approach to Visual Perception. New Jersey, NJ: Lawrence Erlbaum Associates.
Grodal, T. (2009). Embodied Visions: Evolution, Emotion, Culture and Film. New York, NY: Oxford University Press.
Hamilton, J. P., Glover, G. H., Hsu, J. J., Johnson, R. F., and Gotlib, I. H. (2011). Modulation of subgenual anterior cingulate cortex activity with real-time neurofeedback. Hum. Brain Mapp. 32, 22–31.
Hari, R., and Kujala, M. V. (2009). Brain basis of human social interaction: from concepts to brain imaging. Physiol. Rev. 89, 453–479.
Hasson, U., Furman, O., Clark, D., Dudai, Y., and Davachi, L. (2008a). Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding. Neuron 57, 452–462.
Hasson, U., Landesman, O., Knappmeyer, B., Vallines, I., Rubin, N., and Heeger, D. J. (2008b). Neurocinematics: the neuroscience of film. Projections 2, 1–26.
Hasson, U., Malach, R., and Heeger, D. J. (2010). Reliability of cortical activity during natural stimulation. Trends Cogn. Sci. 14, 40–48.
Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., and Malach, R. (2004). Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640.
Heller, A. S., Greischar, L. L., Honor, A., Anderle, M. J., and Davidson, R. J. (2011). Simultaneous acquisition of corrugator electromyography and functional magnetic resonance imaging: a new method for objectively measuring affect and neural activity concurrently. Neuroimage 58, 930–934.
Hoffman, H. G., Richards, T., Coda, B., Richards, A., and Sharar, S. R. (2003). The illusion of presence in immersive virtual reality during an fMRI brain scan. Cyberpsychol. Behav. 6, 127–131.
Hollmann, M., Rieger, J. W., Baecke, S., Lützkendorf, R., Müller, C., Adolf, D., et al. (2011). Predicting decisions in human social interactions using real-time fMRI and pattern classification. PLoS ONE 6:e25304. doi: 10.1371/journal.pone.0025304
Horton, D., and Wohl, R. R. (1956). Mass communication and para-social interaction: observation on intimacy at a distance. Psychiatry 19, 215–229.
Iacoboni, M., Molnar-Szakacs, I., Gallese, V., Buccino, G., Mazziotta, J. C., and Rizzolatti, G. (2005). Grasping the intentions of others with one's own mirror neuron system. PLoS Biol. 3:e79. doi: 10.1371/journal.pbio.0030079
Jääskeläinen, I. P., Koskentalo, K., Balk, M. H., Autti, T., Kauramaki, J., Pomren, C., et al. (2008). Inter-subject synchronization of prefrontal cortex hemodynamic activity during natural viewing. Open Neuroimag. J. 2, 14–19.
Johnston, S. J., Boehm, S. G., Healy, D., Goebel, R., and Linden, D. E. (2010). Neurofeedback: a promising tool for the self-regulation of emotion networks. Neuroimage 49, 1066–1072.
Kaipainen, M., Ravaja, N., Tikka, P., Vuori, R., Pugliese, R., Rapino, M., et al. (2011). Enactive systems and enactive media: embodied human - machine coupling beyond interfaces. Leonardo 44, 433–438.
Kaipainen, M., and Hautamäki, A. (2011). Epistemic pluralism and multi-perspective knowledge organization: explorative conceptualization of topical content domains. Knowl. Organ. 38, 503–514.
Kaipainen, M., Normak, P., Niglas, K., Kippar, J., and Laanpere, M. (2008). Soft ontologies, spatial representations and multi-perspective explorability. Expert Sys. 25, 474–483.
Kaplan, A. Y., Byeon, J.-G., Lim, J.-J., Jin, K.-S., and Park, B.-W. (2005). Unconscious operant conditioning in the paradigm of brain-computer interface based on color perception. Int. J. Neurosci. 115, 781–802.
Kuikkaniemi, K., Turpeinen, M., Laitinen, T., Saari, T., Kosunen, I., and Ravaja, N. (2010). “Influence of implicit and explicit biofeedback in first-person shooter games,” in CHI '10: Proceedings of the 28th International Conference on Human Factors in Computing Systems, (New York, NY: ACM Press), 859–868.
Lahnakoski, J. M., Glerean, E., Salmi, J., Jääskeläinen, I. P., Sams, M., Hari, R., et al. (2012). Naturalistic FMRI mapping reveals superior temporal sulcus as the hub for the distributed brain network for social perception. Front. Hum. Neurosci. 6:233. doi: 10.3389/fnhum.2012.00233
Lee, J. H., Kim, J., and Yoo, S. S. (2012). Real-time fMRI-based neurofeedback reinforces causality of attention networks. Neurosci. Res. 72, 347–354.
Lee, R., Dai, W., and Dix, W. (2010). A decoupled circular-polarized volume head coil pair for studying two interacting human brains with MRI. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2010, 6645–6648.
Lee, R., Dai, W., and Jones, J. (2011). Decoupled circular-polarized dual-head volume coil pair for studying two interacting human brains with dyadic fMRI. Magn. Reson. Med. 68, 1087–1096.
Ma, X., Zhang, H., Zhao, X., Yao, L., and Long, Z. (2012). Semi-blind independent component analysis of fMRI based on real-time fMRI system. IEEE Trans. Neural Rehabil. Eng. doi: 10.1109/TNSRE.2012.2184303. [Epub ahead of print].
Manuck, S. B., Levenson, R. W., Hinrichsen, J. J., and Gryll, S. L. (1975). Role of feedback in voluntary control of heart rate. Percept. Mot. Skills 40, 747–752.
Mar, R. A., and Oatley, K. (2008). The function of fiction is the abstraction and simulation of social experience. Perspect. Psychol. Sci. 3, 173–192.
Mattout, J. (2012). Brain-computer interfaces: a neuroscience paradigm of social interaction? A matter of perspective. Front. Hum. Neurosci. 6:114. doi: 10.3389/fnhum.2012.00114
Montague, P. R., Berns, G. S., Cohen, J. D., McClure, S. M., Pagnoni, G., Dhamala, M., et al. (2002). Hyperscanning: simultaneous fMRI during linked social interactions. Neuroimage 16, 1159–1164.
Moser, E., Derntl, B., Robinson, S., Fink, B., Gur, R. C., and Grammar, K. (2007). Amygdala activation at 3T in response to human and avatar facial expressions of emotions. J. Neurosci. Methods 161, 126–133.
Mori, M. (1970). The Uncanny Valley. Energy 7, 33–35. [Republished in IEEE Robotics and Automation Magazine, June 2012, 98–100].
Neisser, U. (1976). Cognition and Reality: Principles and Implications of Cognitive Psychology. San Francisco, CA: W. H. Freeman and Co.
Nowlis, D. P., and Kamiya, J. (1970). The control of electroencephalographic alpha rhythms through auditory feedback and the associated mental activity. Psychophysiology 6, 476–484.
Parkinson, C., Sinnott-Armstrong, W., Koralus, P. E., Mendelovici, A., McGeer, V., and Wheatley, T. (2011). Is morality unified? evidence that distinct neural systems underlie moral judgments of harm, dishonesty, and disgust. J. Cogn. Neurosci. 23, 3162–3180.
Parsons, T. D., and Courtney, C. G. (2011). “Neurocognitive and psychophysiological interfaces for adaptive virtual environments,” in Human-Centered Design of E-Health Technologies: Concepts, Methods and Applications, eds M. Ziefle and C. Röcker (Hershey, PA: IGI Global), 208–233.
Posse, S., Fitzgerald, D., Gao, K., Habel, U., Rosenberg, D., Moore, G. J., et al. (2003). Real-time fMRI of temporolimbic regions detects amygdala activation during single-trial self-induced sadness. Neuroimage 18, 760–768.
Pugliese, R., and Lehtonen, K. (2011). “A framework for motion based bodily enaction with virtual characters,” in Proceedings of the 10th International Conference on Intelligent Virtual Agents (IVA'11) (Berlin, Heidelberg: Springer-Verlag), 162–168.
Pugliese, R., Tikka, P., and Kaipainen, M. (2012). Navigating story onto space: perspective-relative drive and combinatory montage of cinematic content. Stud. Art Archit. (in press).
Ravaja, N. (2004). Contributions of psychophysiology to media research: review and recommendations. Media Psychol. 6, 193–235.
Redcay, E., Dodell-Feder, D., Pearrow, M. J., Mavros, P. L., Kleiner, M., Gabrieli, J. D. E., et al. (2010). Live face-to-face interaction during fMRI: a new tool for social cognitive neuroscience. Neuroimage 50, 1639–1647.
Rinman, M. L., Friberg, A., Bendiksen, B., Cirotteau, D., Dahl, S., Kjellmo, I., et al. (2004). “Ghost in the Cave - an interactive collaborative game using non-verbal communication,” in Lecture Notes in Computer Science, eds A. Camurri and G. Volpe (Berlin: Springer-Verlag), 549–556.
Riva, G., Raspelli, S., Algeri, D., Pallavicini, F., Gorini, A., Wiederhold, B. K., et al. (2010). Interreality in practice: bridging virtual and real worlds in the treatment of posttraumatic stress disorders. Cyberpsychol. Behav. Soc. Netw. 13, 55–65.
Rizzo, A. A., Schultheis, M. T., Kerns, K., and Mateer, C. (2004). Analysis of assets for virtual reality applications in neuropsychology. Neuropsychol. Rehabil. 14, 207–239.
Rizzolatti, G., and Fabbri-Destro, M. (2008). The mirror system and its role in social cognition. Curr. Opin. Neurobiol. 18, 179–184.
Sanchez-Vives, M. V., and Slater, M. (2005). From presence to consciousness through virtual reality. Nat. Rev. Neurosci. 6, 332–339.
Sarkheil, P., Zilverstand, A., Kilian-Huetten, N., Schneider, F., Mathiak, K., and Goebel, R. (2011). “Application of real-time fMRI-neurofeedback in treatment of emotional disorders,” in Abstract at 17th Annual Meeting of the OHBM (Quebec, QC: Canada).
Saygin, A. P., Chaminade, T., Ishiguro, H., Driver, J., and Frith, C. (2011). The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions. Soc. Cogn. Affect. Neurosci. 7, 413–422.
Singer, T., Seymour, B., O'Doherty, J. P., Stephan, K. E., Dolan, R. J., and Frith, R. J. (2006). Empathic neural responses are modulated by the perceived fairness of others. Nature 439, 466–469.
Tan, E. S. (1996). Emotion and the Structure of the Narrative Film. Hillsdale, NJ: Lawrence Erlbaum.
Tikka, P. (2005). Enactive Cinema Installation “Obsession” Exhibited in the Museum of Contemporary Art Kiasma. Helsinki: Oblomovies and University of Art and Design.
Tikka, P. (2008). Enactive Cinema: Simulatorium Eisensteinense. Ph.D. dissertation, Helsinki: University of Art and Design Publication Series.
Tikka, P., Vuori, R., and Kaipainen, M. (2006). Narrative logic of enactive cinema: ‘Obsession’. Digital Creativity 17, 205–212.
Varela, F., Thompson, E., and Rosch, E. (1991). Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press.
Veit, R., Singh, V., Sitaram, R., Caria, A., Rauss, K., and Birbaumer, N. (2011). Using real-time fMRI to learn voluntary regulation of the anterior insula in the presence of threat-related stimuli. Soc. Cogn. Affect. Neurosci. 7, 623–634.
Voyvodic, J. T., Glover, G. H., Greve, D., and Gadde, S. (2011). Automated real-time behavioral and physiological data acquisition and display integrated with stimulus presentation for FMRI. Front. Neuroinform. 5:27. doi: 10.3389/fninf.2011.00027
Weiskopf, N., Mathiak, K., Bock, S. W., Scharnowski, F., Veit, R., Grodd, W., et al. (2004). Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI). IEEE Trans. Biomed. Eng. 51, 966–970.
Keywords: enactive cinema, real-time fMRI, neurofeedback, social neuroscience, generative storytelling, implicit interaction, Brain Computer Interfaces
Citation: Tikka P, Väljamäe A, de Borst AW, Pugliese R, Ravaja N, Kaipainen M and Takala T (2012) Enactive cinema paves way for understanding complex real-time social interaction in neuroimaging experiments. Front. Hum. Neurosci. 6:298. doi: 10.3389/fnhum.2012.00298
Received: 29 February 2012; Accepted: 11 October 2012;
Published online: 01 November 2012.
Edited by:
Kai Vogeley, University Hospital Cologne, GermanyReviewed by:
Gary Bente, University of Cologne, GermanyRobert P. Spunt, California Institute of Technology, USA
Copyright © 2012 Tikka, Väljamäe, de Borst, Pugliese, Ravaja, Kaipainen and Takala. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.
*Correspondence: Pia Tikka, Department of Film, Television and Scenography, Aalto University School of ARTS, P.O. BOX 31000, 00076 Aalto, Finland. e-mail: pia.tikka@aalto.fi