The Movement-Image Compatibility Effect: Embodiment Theory Interpretations of Motor Resonance With Digitized Photographs, Drawings, and Paintings

To evoke the impression of movement in the “immobile” image is one of the central motivations of the visual art, and the activating effect of images has been discussed in art psychology already some 100 years ago. However, this topic has up to now been largely neglected by the researchers in cognitive psychology and neuroscience. This study investigates – from an interdisciplinary perspective – the formation of lateralized instances of motion when an observer perceives movement in an image. A first step was to identify images that evoke a perception of movement in a certain direction and to give this a rating. Reaction times leading to the engagement of a joystick following the presentation of images are used to evidence the postulated movement occasioned by the perception of movement in an image. Where the required direction of joystick moves matched the expected perception of movement direction in the image, significantly shorter reaction times were recorded. The experiment was able to prove a “movement-image compatibility effect” in observers of images. Based on this, the paper revisits and brings up to date the theses on motor sensory response to images which were developed in art psychology at the beginning of the 20th century. It furthermore contributes an embodiment theory interpretation to the prevalent representational explanation of compatibility effects.

Introduction

In 1922 Paul Klee asked, “What does movement in the work actually mean? Our art is not capable of movement, is it? We are not fabricators of automatons, after all! No. Our works will by their very nature be confined to one place and stay there. And yet they are all about movement” (Transl. by the author, Klee and Glaesemer, 1979, p. 94). To evoke the impression of movement in the “immobile” image is one of the central motivations of the visual arts; at the same time, it is a seemingly impossible task. In the interaction between artist and image, but also in the processes of perception between image and observer, the body and its potential for movement seems to play a vital role. The present study has been developed by an interdisciplinary group and is dedicated to the examination of this subject.

In art psychology, research of the processes that are involved in the perception of movement in the image is a central concern (e.g., Arnheim, 1966). It is one to which cognitive psychology and neuroscience have generally paid little heed, despite the fact that insights into the activating effect of images have for some time been available (Lauschke, 2014, 2016, 2017a,b). In art history, efforts to understand the activating effect of images led to the formation of a “picture act” theory, which understands images as agents (Bredekamp, 2010). In recent years eye-tracking studies have been used in connection with the observation of works of art and for the first time art historical research was combined with psychology and cognitive science (for example, Rosenberg and Klein, 2015). Our project aims to expand empirical image studies by providing more information on the role of the entire body and its motion response to image perception.

Our study is a contribution to empirical investigations related to the issue of “embodied cognition.” Many experiments, using diverse methods, have been carried out in the pursuit of various phenomena in the same context. Such phenomena include: linguistic competence (Barsalou et al., 2003; Pulvermüller et al., 2004; Louwerse, 2008; Rueschemeyer et al., 2010; Glenberg and Gallese, 2012; Pulvermüller, 2013), the perception of space (Holmes and Spence, 2004; Cardinali et al., 2009a), multi-modal integration (Sirigu et al., 1991; Maravita et al., 2003), the use of instruments (Cardinali et al., 2009b) and the perception of objects (Tucker and Ellis, 1998). Experimental work has, however, been cautious in tackling the subject of image perception. To date, there are few publications in this field (Freedberg and Gallese, 2007; Proverbio et al., 2009; Umiltà et al., 2012; Sbriscia-Fioretti et al., 2013; Ferretti, 2016).

This study does not limit itself to a “moderate embodied cognition approach” (Goldman, 2012). It proceeds on the premise of a “full-blown” embodiment theory, which argues that even the non-neural parts of a body contribute to the constitution of cognitive conditions and processes. Such a premise gives rise to certain methods. If cognition in general and thus also perception – or the perception of movement in the image – are to be understood as integral to the physical equipment of a cognitive organism, then empirical embodiment theory studies must take the entire corporeality of a cognitive organism into account, in order for questions on specific cognitive abilities to be answered. Neuroscientific methods (such as EEG or fMRT) require the subject to be immobile. This inhibits the very motor resonance between the perception and performance of movement which is at the heart of our study. For this reason, we decided to use reaction time measures in our study.

A further consequence of the above-mentioned embodiment theory premise is that interdisciplinarity becomes an essential factor in addressing questions of cognitive science. For example, in cognitive neuroscience, the body must play a subordinate role since cognitive neuroscience focus on neural structures and activities. Yet in studies that explicitly examine the body, such as biology or human anatomy, the requirement is mostly absent that cognitive phenomena be explained through research findings (Shapiro, 2004). A challenge therefore arises: results from various disciplines must be combined in such a way as to give credence to cognitive science theories without restricting evidence to that found in neural networks and activities.

In cognitive psychology, “compatibility effect” is a term describing, among other things, the phenomenon whereby the location of a stimulus co-determines the type of response to it (Umiltà and Nicoletti, 1990). According to Kornblum et al. (1990), this can be perceived as resulting from overlapping spatial dimensions of the physical world and of movement dimensions of the behaving body (Dimensional Overlap Model, DOM). In experiments it has been found that in situations in which the location of a stimulus and the direction of required action are congruent, reaction times are shorter than in the opposite case. The phenomenon has come to be known as the “Simon effect” (Simon, 1990) and it is conceptualized by Kornblum et al. (1990) proposing the DOM. A similar effect was observed by Glenberg and Kaschak (2002) in respect of motor reactions to spoken stimuli (“action-sentence compatibility effect”).

In the theoretical contexts of art psychology and esthetics, theories were developed between 1880 and 1930 which may be thought of as preparing the way for today’s research into compatibility effects. It should be stressed, however, that those earlier theories were formulated from a different perspective than current research into “compatibility effects.” Whilst original concerns were directed to understand the esthetic experience of images, cognitive psychology seeks to investigate effects on behavior. In the late 19th century, Wölfflin established a link between art history and psychology. He brought the human body into focus in its significance for the perception of art. Wölfflin (1886) formulated what was for his times a daring hypothesis, assuming that visual stimuli are directly conducted to the motor apparatus. Only much later would this be adopted by cognitive psychology. One source of his ideas came from the esthetics of empathy (Einfühlungstheorie). According to Vischer (1873), mental agitation in perception was apparent in physical movement. Vischer explained “empathetic motor response” as a factor of the close interdependence of the visual and tactile senses. Lipps (1923, p. 144) maintained that forms prompted impulses toward movement in the observer. He also made clear that the same vitality that observers notice in their own or other human bodies could be ascribed to figures in images. From our current perspective, the cited work of Wölfflin, Vischer, and Lipps not only resembles the assumptions of embodiment theory but may also be understood as a collection of pioneering theories in the field of “compatibility effects.”

Laboratory findings by Münsterberg (1889), a pioneer in applied and experimental psychology, confirm the presumption of a direct connection between sensory and motor systems, as well as of the existence of involuntary bodily responses to perceived stimuli. The art psychologist and philosopher Müller-Freienfels drew on Münsterberg’s findings (1922). He coined the term “motor resonance” to capture instances of movement in response to the perception of movement in the image on the part of a beholder. His criticism, now some 100 years old, of “psychological intellectualism that seeks to explain everything by representation” (Transl. by the author, Müller-Freienfels, 1922, p. 118) has in the last three decades become a subject of debate once more, following a turn toward the body in philosophy and in the cognitive, art and cultural sciences. The term “motor resonance” is today applied in neuroscience (Uithol et al., 2011) and in cognitive psychology (Schütz-Bosbach and Kuehn, 2014) – albeit without reference to Müller-Freienfels. Müller-Freienfels’ “motor resonance” and Münsterberg’s efforts have opened a horizon in theory that allows a very close connection between perception and body, between sensory and motor systems, and between stimulus and response (such as occurs in connection with compatibility effects) to be substantiated.

In the 1990s the working group of Odmar Neumann in Bielefeld attempted to prove the potential for direct motor activation through visual stimuli independently of conscious visual perception (“response priming”) and to confirm Münsterberg’s hypotheses using modern experimental psychology and electro-physiology (Ansorge et al., 1998; Neumann et al., 1998; Klotz and Neumann, 1999; Ansorge and Leder, 2011; see also Schubert et al., 2013). The common coding theory developed by Prinz and others (Prinz, 1990) and Hommel’s event coding theory (Hommel et al., 2001) are based on the assumption that perception and action are directly linked via a common code. This theory suggests that motor functions are occasioned by the mental representation of the effects of anticipated action. Both theories refer to stimulus-response compatibility and explain this by proposing that stimulus information releases “lateralized readiness potential” and even gives rise to subliminal movements associated with the presented stimulus (Eimer et al., 1995; Hommel, 1997).

Freedberg and Gallese (2007) were the first to employ neuroscientific methods to investigate bodily reactions to images. They posited that the motive effect of images can be attributed to bodily resonance and results from the activation of embodied mechanisms, i.e., neural-based automatic simulation of actions, emotions and bodily sensations. Proverbio et al. (2009) showed that photographs portraying intensely dynamic actions possessed a strong cortical activation potential, compared with less dynamic images. Ferretti (2016) argues that “action properties” may be attributed even to objects portrayed within images. In 2012 a research group examined whether looking at abstract art could release motor resonance (Umiltà et al., 2012). Using EEG, measurements were taken of the changes in brain waves of beholders of works by the Italian artist Lucio Fontana. Significant suppression of μ-rhythms was observed in beholders of Fontana’s work, with its slashed canvases. No suppression of μ-rhythms was observed in control experiments using graphically modified versions of Fontana’s work. The authors of the study drew the conclusion that looking at the slashed canvas prompted the same activation of the motor cortex as would be triggered in a person observing, or performing the action of slashing. However, it should be noted that the reactions to Fontana’s images cannot be generalized in respect of abstract art; the peculiarity of his images consisting in the fact that the slashes are “real” rather than “abstract.”

The present study is linked to the investigation on image perception and refers to the assumptions that have emerged over the course of some 100 years now accepting the existence of motor resonance between images and their beholders. The aforementioned premise of the study, based on embodiment theory, will take this pioneering work into account. We pursue a “full-blown” embodied cognition approach that is, on the one hand, not only historically contextualized but also theoretically supported by art historians (Bredekamp, 2010) as well as philosophers (Gallagher, 2005; Krois, 2011). On the other hand, it is embedded in empirical research that uses different methods and experimental designs than commonly applied for studying similar issues. By conducting the study in this way, we hope to shed new light on the issue of image perception. The experiment set-up borrows from a study conducted by Glenberg and Kaschak (2002) in order to develop the line of empirical and embodied cognition research further. For that purpose, we aimed to measure an image-compatibility effect, when participants observe digitized photographs, drawings, and paintings that show movement. In particular, we asked participants to judge whether or not an image involved a movement impression by moving a joystick to the left or to the right side. Subsequently, we divided the presented images in images with a left- or right-sided movement direction and analyzed the reaction times of participants depending on the compatibility relation between the yes or no-movement response and the direction of the movement in the presented image (i.e., left or right-sided movement direction). We expected decreased response times in compatible conditions (i.e., conditions in which the side of the yes-movement response coincided with the movement direction of the presented image) compared to incompatible conditions. In the first step of the investigation, we identified images that evoke a perception of movement. In the second step of the investigation, we measured the reaction times from the presentation of images to the engagement of a joystick (“joystick move”) (Figure 1).

FIGURE 1

FIGURE 1. The experiment on the perception of movement in images. Participants were asked to indicate by moving a joystick whether they perceived a suggestion of movement in the image (IIa,b) or not (I). Shorter reaction times were anticipated if the required joystick move was compatible with the perceived direction of movement (IIa) than in the contrary scenario (IIb).

Materials and Methods

Participants

All procedures performed in the experiment were in accordance with the standards of the ethics committee of the Humboldt-Universität zu Berlin and with the 1964 Helsinki declaration and its later amendments. The ethics committee of the Humboldt-Universität zu Berlin approved the study. Written informed consent was obtained from all participants. Separately, 25 people took part in the rating of selected images (14 women, 11 men) with an average age of 24.7 years (SD = 4 years; ages in the range of 18–36 years). The experiment itself involved 18 participants (8 women, 10 men) with an average age of 25.2 years (SD = 2.1 years, ages in the range of 21–29 years). All participants were recruited from the participant database of the Department of Psychology at Humboldt-Universität zu Berlin, were registered and received eight Euro per hour for compensation. Prior to the experiment, a standardized handedness test was used to ensure that all participants were right-handed (Oldfield, 1971). The participants passed the test with a mean laterality quotient of 71.72 ± 23.21.

Image Selection

Image selection was performed on the basis that an image either evoked the perception of movement that was specifically directed toward the left or right (edge of the image) or that it gave the impression of a resting position – in other words, these images did not suggest any direction of movement. Photographs, paintings and drawings from a variety of art periods were considered. Most images depicting movement contained characteristics of a posture and position of a moving figure (humans and/or animals), such as a rolling movement of the foot, shifting of weight or the impression of a movement about to be, or having been undertaken as well as the positioning (of the body) in the picture and the orientation of a figure and direction of gaze, including in relation to composition or perspective. The following indicators of movement were chosen: (i) “Fluid” clothing. The art historian Aby Warburg developed the term “bewegtes Beiwerk” hence the undulation of hair and clothes enhances the impression of an inner motion of a figure (see Figure 2A as example). (ii) Superimposition of single sequences of movement (such as in multiple exposure images). The superimposition of single sequences of movement plays a crucial role for evoking motion as it was already developed in chronophotographic studies in 19th century (see Figure 2B as example). (iii) Traces of movement such as fuzziness, blurring, lines (see Figures 2C,D as examples). An image might fulfill several of these criteria. In the example of the “Derby of Epsom” by Théodore Gericault (1821) some of these criteria can be seen, even if the running horses anatomically are not correct (Figure 2E).

FIGURE 2

FIGURE 2. Examples for images with perceived movement. (A) “Bewegtes Beiwerk”, i.e., apparently moving clothes or hair. (B) Chronophotographic image sequence. (C,D) Motion blur. (E) “The Derby of Epsom” by Théodore Gericault.

The images were then evaluated by independent observers on a scale from -5 (perceived movement to the left) to +5 (perceived movement to the right), with the “0” reserved for images that did not suggest movement of any kind. 16 images were selected that had been evaluated at “0” by every observer and were used as control images (hereafter “static images”). Fifteen images received an average rating value of more than +2.8 or less than -2.8 and were subsequently included in the experiments. The results of the rating show that mostly images with figural motifs were able to evoke a distinct impression of movement toward a certain direction in the observer. Abstract motifs, even if demonstrating specific directions of movement or a tendency within the composition format through arrows or lines, were insufficient to produce the same impression. However, it should not be concluded that abstract images are in principal unable to evoke the impression of movement. It might well be that the specific questions of the rating made it hard for abstract images to be considered for the experiment.

Apparatus and Stimuli

The 15 images with the highest average evaluation of perceived movement were each reproduced as horizontal mirror images. The data set thus comprised 30 images with perceived movement and was balanced in respect of the direction of perceived movement and intensity of perceived movement in a direction. Together with the 16 static images this made a final data set of 46 stimulus images. The experiment was programmed using “Presentation” (Version 17.0, Neurobehavioral Systems, Inc. 2016). All 46 images were produced in a similar size of approximately 20 × 20 cm and presented with a central viewing distance of 70 cm against a dark background (CIE: x = 0.313, y = 0.329; luminance = 12.46 cd/m²). The screen was a 24 inch LED-TFT monitor with a resolution of 1920 × 1080 pixels and a refresh rate of 100 Hz.

Design and Procedure

The 30 movement images and 16 static images were presented in a random sequence, in two blocks (i.e., 92 images were presented to each participant). The participant’s gaze was directed with the help of a small cross in the center of the screen. With the eye fixated during 500 ms, each image was presented centrally in front of the participant for 150 ms. Following reaction to the image, an empty dark screen was shown for 5000 ms during the inter trial intervals. The participants trained six exercise trials before the experiment. This was followed by the first block and, after a short break, the second block. The participants were asked to use the joystick to indicate if they had perceived any movement in the image and to do so as fast and correct as possible. For half of the participants, a joystick movement to the left was chosen to indicate positive confirmation, i.e., movement present (group A) and a right movement indicated no movement present in the image. For the other half of the participants, the response assignment was reversed (group B). Reaction times were measured from the presentation of the image to the first measurable move of the joystick. The direction of the first measurable move of the joystick was also registered.

Statistical Analysis

A total of 1,656 reaction times (46 images × 2 blocks × 18 participants) were collected. A linear mixed model was calculated and depicted using the free statistics software “R” in “R Studio” (R statistics version 3.3.2; R Development Core Team (2018)) using several packages (plyr, lme4, ggplot2, effects, and car). The model was used to check the data set for the postulated movement-image compatibility effect. In addition to the hypothesized effect of the compatibility of the expected perception of movement in the image and the reaction times of participants, we included the number of completed trials (for “repeated exercise” effects) as fixed effects in the model. Because joystick movements of the right handed participants generally were faster to the left independent from the perceived movement in the images (cf. Figure 3), we treated both groups (A and B) separately to study the hypothesized effect of movement-image compatibility. Joystick movements that did not correspond to expectations were classified as errors and removed from the analysis. However, it should be emphasized that an unexpected direction in joystick movements might be explained by a corresponding perception in the participant, and therefore potentially did not constitute an error. Participant No. 7 was excluded from the analysis due to extremely high variance in reaction times.

FIGURE 3

FIGURE 3. Reaction times in groups A and B for the perception of movement in the image as confirmed by the joystick move. Note that in group A participants were asked to indicate a perceived movement by moving the joystick to the left and in group B participants were asked to indicate perceived movement by moving the joystick to the right. In both groups, reaction times were on average shorter if the anticipated direction of perceived movement in the image was the same as that of the required joystick move (i.e., in the case of congruent directions).

The following steps were carried out when analyzing the statistics. First, the “R” function “lmer” was used to establish a linear mixed model, which is suited to account for fixed and random effects. The model had the following formula (explained below):

$\mathrm{\text{log}}\mathrm{\left(}\mathrm{R}\mathrm{T}\mathrm{\right)}\mathrm{\sim }\mathrm{\text{poly}}{\mathrm{\left(}\mathrm{\text{Trial, 3}}\mathrm{\right)}}^{\mathrm{*}}\mathrm{\text{Block}}\mathrm{+}{\mathrm{\text{Movement}}}^{\mathrm{*}}\mathrm{\text{Reaction+}}\mathrm{(}\mathrm{1}\mathrm{+}\mathrm{\text{Reaction+}}\mathrm{\text{Movement|}}\mathrm{\text{participant}}\mathrm{)}\mathrm{+}\mathrm{\left(}\mathrm{1}\mathrm{\right|}\mathrm{\text{image}}\mathrm{)}$

Since normal distribution was not given in the raw reaction times, normal distribution of data was subsequently achieved using logarithms of reaction times [the dependent variable log(RT)]. On the left side of the tilde (∼) the independent variables are coded. The term “poly(Trial,3)^∗Block” in fixed effects is a third grade polynomial that approximates the development of reaction times over the duration of the experiment; i.e., the “repeated exercise” effect. “Movement^∗Reaction” is the expected movement-image compatibility effect (i.e., a fixed effect, too). The model also takes into account random effects attributable to the individuality of the participants (the term 1 + Reaction + Movement| participant) as well as the variation in strengths of perception of movement when looking at individual images (term 1| image).

Besides normal distribution which was achieved by using logarithmized reaction times and to further verify the data structure preconditions (i.e., the assumptions) of the mixed model, a residual plot was produced (not shown). Residuals with values above 0.5 were removed (elimination of outliers). Similarly, steps were taken to ensure there was homoscedasticity of variance.

Finally, significance was checked using a log-likelihood based Chi² test (χ²). This involved the exclusion of individual terms from the model; a comparison of the resulting model was made to establish which term had a significant effect. Furthermore, a confidence interval (CI) was produced.

Results

Of the total number of 1,564 registered joystick movements (after the exclusion of participant No. 7) only 45 failed to comply with the expected direction of reaction based on the rating – which corresponds to an error rate of approximately 2.9% (on average 2.65 ± 3.44 errors out of 92 required reactions per participant). In 18 cases a perceived movement was confirmed by the joystick even though the rating had determined that the image in question did not evoke a perception of movement. In 27 cases, the joystick move indicated that there was no movement perceived in the image even though the stimulus was considered to be an image evoking the perception of movement. In only seven of these 27 cases (0.4% of all required reactions) the joystick was moved in a direction other than that determined by the rating.

Consistent with the expectation, both groups (A and B) responded faster in the compatible condition (perceived movement in the image and required direction of joystick movement are congruent) than in the in-compatible condition (Figure 3). This observation was confirmed by the linear mixed model (Figure 4). Logarithms of reaction times were significantly smaller in the compatible condition (χ² = 4.71; p = 0.03) than in the in-compatible condition (CI range from 2.5%: -0.0257 to 97.5%: -0.0015).

FIGURE 4

FIGURE 4. Effect plot of the interaction in the mixed model between the direction per expected perception of movement in the image as established in the rating and the required direction of the joystick move in confirmation plotted for groups A and B separately. Logarithmised reaction times [log(RT)] were used for the model. The model yielded a significant effect of interaction.

The duration of the experiment also had a significant influence on the result (χ² = 17.70; p < 0.001). A repeated exercise effect was thus confirmed over the course of the experiment.

Discussion

The aim of this study was to use the measurement of reaction times from presentation of an image to joystick move to prove an instance of physical movement caused by the perception of movement within an image. The primary finding of this experiment was the so-called “movement-image compatibility effect.” Analysis of the reaction times shows that participants require more time to give their answers where the perceived movement within the image is incompatible with the movement of the joystick required to give the answer (cf. response priming: Fehrer and Raab, 1962; Rosenbaum and Kornblum, 1982). The phenomenon involved here is also known as “motor resonance”. The following will discuss the results. The discussion will center on two possible interpretations of the results, both of which refer to the tradition of embodiment cognition research.

Multi-Modal Representations and Priming

The stimulus-reaction pattern found in the experiment could be explained by focusing on the neurophysiological structure and its activity that is involved in the translation of stimulus into reaction. The claim is that perceptive input is neurally represented and processed by the activity of networks of neurons that are also partly responsible for preparing and controlling motor output (see DeCharms and Zador, 2000 on the term “neural representation”). A variation in reaction times to given stimuli is thus to be expected. The simultaneous representation, processing and preparation of both stimulus information and required motor reaction to the stimulus leads to interference in the cognitive systems of recipients, which in turn leads to longer reaction times. Attempts to explain compatibility effects in this way are often based on the assumption that both sensory stimuli and motor reactions are represented by internal, multi-modal, and neural representations; see “bodily formatted representations” (Goldman and de Vignemont, 2009; Goldman, 2012), “grounded cognition” (in Barsalou et al., 2003), or the “common coding theory” (Eimer et al., 1995). Bodily formatted representations (henceforth B-formatted representations) are neurally constituted, body-specific representations. They contain somatosensory, affective, interoceptive and motor-related information; they are multi-modal representations. Presumably, stimulus processing partly depends on such information. Hence, motor actions and sensory attributes might well be linked to each other via this kind of neural representation. For this to be true, it would mean that the neural components of the brain that generate body-specific representations also contribute to the execution of perception processes.

In the context of Goldman’s “moderate embodied cognition” approach and the idea of B-formatted representations, the neurophysiological parts one would expect to be involved in bringing about the movement-image compatibility effect are, on the one side, the visual cortex in the occipital lobe (Brodmann areas 17, 18, 19) since these neural areas are responsible for visual processing. However, because the proposal is that visual information might also be represented by the activity of motor-related neural networks, the primary motor, premotor, and supplementary motor cortex (Brodmann area 4, 6, and 44) (Pulvermüller, 2013) are very likely to contribute to the processing of visual stimuli and hence to play a vital role for constituting the movement-image compatibility effect, too. Since it is assumed that B-formatted representations are distributed over the networks of mirror neurons, they are brought about by similar neural structures as mirror effects are (Goldman, 2012). Mirror effects relate to phenomena whereby the mere observation of an action results in activation of those neural areas that would be required to carry out the observed action (Keysers et al., 2003; Avenanti et al., 2007; Keysers, 2009). Mirror effects are evoked by so-called mirror neurons (Rizzolatti et al., 1996; Keysers et al., 2010) and they are sometimes understood as a phenomenon of neural reprocessing or redeployment (Anderson, 2008, 2010; Goldman, 2012). The core networks of mirror neurons in humans are, according to Rizzolatti and Craighero (2004, p. 176) formed by “occipital, temporal, and parietal visual areas, […] the rostral part of the inferior parietal lobule and the lower part of the precentral gyrus plus the posterior part of the inferior frontal gyrus.”

Altogether, the concept of B-formatted representation links the sensory and motor systems in such a manner that experiments investigating the perception of movement in the image, such as those described here, evoke interference in the cognitive system of the recipient. This leads to prolonged reaction times, particularly in cases where a given stimulus activates the sensory-motor neuron system in such a way that the participant must first overcome an activation impulse in order to execute the target reaction.

Alternative Interpretations: Movement-Image Compatibility as Body Schematic Response

For some 30 years, a debate has been developing on the topic of “situated cognition” involving various disciplines such as psychology, philosophy of mind, neuroscience and artificial intelligence studies. Current research of situated cognition has coined such terms as “extended mind,” “embedded cognition,” “enactivism” and “embodiment theory.” Some theories behind those terms have in common that they reject today’s most dominant cognitivist view that cognition is an input-output conversion that can be couched in representational terms (e.g., Hutto and Myin, 2013, 2017). A specific group of approaches among those critical theories emphasize the constitutive role of the body, or non-neural parts of the body for cognitive processes. They focus on a special understanding of the “body schema” in order to understand how cognitive processes like visual perception arise (Gallagher, 2001, 2005, 2012, 2015; Krois, 2011). The extended reaction times to given stimuli that were observed in the experiment might be explained by a “full-blown embodied cognition” approach that refers to the body schema of participants that is affected by the perception of movement in the image.

The term “body schema” was prominently introduced by the neurologist Head (Head and Holmes, 1911), who saw it as a model of physical posture. Nowadays, it is still used in sensorimotor cognition theories (for an overview of these, see for instance Bishop and Martin, 2014). It is important to stress that there exist several versions of the concept of the body schema (Cardinali et al., 2009a). In many studies, aspects of the body schema (such as body ownership or peripersonal space) are cited in the investigation of sensory and multi-sensory processes (Makin et al., 2008; Ehrsson, 2012). Other studies identify the “self” with the body schema (de Vignemont, 2007) and examine aspects of a person’s consciousness, among other things. The conception proposed in the following refers to the analysis of (multi-) sensory processes. It does not necessarily accept representationalist theories which in several sources are linked to the body schema (e.g., Makin et al., 2007). Krois (2011) develops this term and redefines it as a subpersonal, multi-sensory system that automatically controls posture and movement. It governs reflexes, physical progression, instrumental movements such as grasping, and expressive movements such as gestures (Gallagher, 2005); it therefore also governs the joystick moves that were required of participants in the experiment. The body schema is not identifiable as a single part of the body. It is holistic because it is the coordination of information won from diverse parts of the body and from diverse sensory processes. It coordinates visual, haptic and proprioceptive information such that, for instance, bodily balance can be maintained. In order to conduct the coordination of sensorimotor processes, the body schema must fulfill two functions. It must integrate retentional as well as protentional aspects of movement. A change of position just carried out (the so-called retentional aspect of an actual movement) influences which motor procedures can be subsequently performed (the protentional aspect of movement). The coordination of retentional and protentional aspects of motor procedures are made possible through the integration of diverse sensory information. Krois posited that images give rise to the perception of dynamic relationships as vital forces. Bredekamp (2010) investigated the interconnection of embodiment and the activating effect of images. The claim is that processes of the body schema such as the maintenance of balance, the understanding of spatial relations, and the implicit knowledge of one’s own ability to move are also at work when observing an image. In the case of perception of images that show movement, the visual information is treated by the body as motor information; so a motor disposition is built up in recipients on a subpersonal level that puts the observer in a state of readiness. This leads to extended reaction times in those cases where the required joystick move in the experiment does not conform to the direction of movement in the image.

Conclusion

Our experiment on the perception of movement in images was able to show evidence for the motor resonance of images. The proven effect is called “movement-image compatibility effect.” The experiment also provided further confirmation that the effects and perception of images nonetheless remain complex and challenging areas. As a result of the rating conducted as part of the selection of images and before the experiment itself, it became apparent that mostly figural motifs had been identified as suggesting movement. In the rating, images with a more complex structure such as abstract motifs were insufficient to produce a marked impression of movement in the observers. This is also the case if they demonstrate specific directions of movement or at least tendencies within the composition format through arrows or lines. Future research should aim to explain these circumstances, which are far from being fully understood.

The discussion on a possible explanation of the movement-image compatibility effect and similar effects furthermore showed that various disciplines have put forward consistent hypotheses with arguments at diverse levels. It is not possible to come to a conclusion as to whether significantly longer reaction times in the experiment presented here could be caused by neural interference, by multi-sensory integration or by a concatenation of both. However, the term “body schema” allows us to include the entire biological equipment of a cognitive organism, and not just its neurophysiological structures, when investigating cognitive abilities such as perception or the perception of movement in the image. This does not mean that experimental research into cognition can rely solely on the testing of behavioral hypotheses. The brain plays an important role in the constitution of cognitive states and processes. Only through neuroscientific methods can the details of brain functions or areas of the brain be discovered and understood. The challenge is in integrating the results in a broader debate about what constitutes cognitive states and processes like visual perception of images.

Author Contributions

AP, JAN, M-OC, and ML contributed conception and design of the study. AP, JN, M-OC, and ML organized the database. JAN performed the statistical analysis. TS reviewed the statistical analysis. CR and TS enabled the performance of the experiment. AP, JAN, M-OC, and ML wrote sections of the manuscript. CR and TS revisited and commented the manuscript. All authors contributed to manuscript revision, read and approved the submitted version.

Funding

The study received funding from the German Research Council (grant number DFG EXC 1027 “Image Knowledge Gestaltung. An Interdisciplinary Laboratory”).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

Anderson, M. L. (2008). Circuit sharing and the implementation of intelligent systems. Conn. Sci. 20, 239–313. doi: 10.1080/09540090802413202

CrossRef Full Text | Google Scholar

Anderson, M. L. (2010). Neural reuse: a fundamental organizational principle of the brain. Behav. Brain Sci. 33, 245–266. doi: 10.1017/S0140525X10000853

PubMed Abstract | CrossRef Full Text | Google Scholar

Ansorge, U., Klotz, W., and Neumann, O. (1998). Manual and verbal responses to completely masked (unreportable) stimuli: exploring some conditions for the metacontrast dissociation. Perception 27, 1177–1189. doi: 10.1068/p271177

PubMed Abstract | CrossRef Full Text | Google Scholar

Ansorge, U., and Leder, H. (2011). Wahrnehmung und Aufmerksamkeit. Wiesbaden: Verlag für Sozialwissenschaften, doi: 10.1007/978-3-531-92094-8

CrossRef Full Text | Google Scholar

Arnheim, R. (1966). “Perceptual and aesthetic aspects of the movement response,” in Toward a Psychology of Art: Collected Essays, ed. R. Arnheim (Berkeley, CA: University of California Press), 74–89. doi: 10.1111/j.1467-6494.1951.tb01101.x

CrossRef Full Text | Google Scholar

Avenanti, A., Bolognini, N., Maravita, A., and Aglioti, S. M. (2007). Somatic and motor components of action simulation. Curr. Biol. 17, 2129–2135. doi: 10.1016/j.cub.2007.11.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Barsalou, L. W., Simmons, W. K., Barbey, A. K., and Wilson, C. D. (2003). Grounding conceptual knowledge in modality-specific systems. Trends Cogn. Sci. 7, 84–91. doi: 10.1016/S1364-6613(02)00029-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Bishop, J. M., and Martin, A. O. (2014). Contemporary Sensorimotor Theory, Vol. 15. London: Springer Science+Business Media. doi: 10.1007/978-3-319-05107-9

CrossRef Full Text | Google Scholar

Bredekamp, H. (2010). Theorie des Bildakts: Frankfurter Adorno-Vorlesungen 2007. Berlin: Suhrkamp Verlag.

Google Scholar

Cardinali, L., Brozzoli, C., and Farnè, A. (2009a). Peripersonal space and body schema: two labels for the same concept? Brain Topogr. 21, 252–260. doi: 10.1007/s10548-009-0092-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Cardinali, L., Frassinetti, F., Brozzoli, C., Urquizar, C., Roy, A. C., and Farnè, A. (2009b). Tool-use induces morphological updating of the body schema. Curr. Biol. 19, R478–R479. doi: 10.1016/j.cub.2009.05.009

PubMed Abstract | CrossRef Full Text | Google Scholar

de Vignemont, F. (2007). Habeas corpus: the sense of ownership of one’s own body. Mind Lang. 22, 427–449. doi: 10.1111/j.1468-0017.2007.00315.x

CrossRef Full Text | Google Scholar

DeCharms, R. C., and Zador, A. (2000). Neural representation and the cortical code. Annu. Rev. Neurosci. 23, 613–647. doi: 10.1146/annurev.neuro.23.1.613

CrossRef Full Text | Google Scholar

Ehrsson, H. H. (2012). “The concept of body ownership and its relation to multisensory integration,” in The New Handbook of Multisensory Processing, ed. B. E. Stein (Cambridge: The MIT Press), 775–792.

Google Scholar

Eimer, M., Hommel, B., and Prinz, W. (1995). S-R compatibility and response selection. Acta Psychol. 90, 301–313. doi: 10.1016/0001-6918(95)00022-M

CrossRef Full Text | Google Scholar

Fehrer, E., and Raab, D. (1962). Reaction time to stimuli masked by metacontrast. J. Exp. Psychol. 63:143. doi: 10.1037/h0040795

CrossRef Full Text | Google Scholar

Ferretti, G. (2016). Pictures, action properties and motor related effects. Synthese 193, 3787–3817. doi: 10.1007/s11229-016-1097-x

CrossRef Full Text | Google Scholar

Freedberg, D., and Gallese, V. (2007). Motion, emotion and empathy in esthetic experience. Trends Cogn. Sci. 11, 197–203. doi: 10.1016/j.tics.2007.02.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Gallagher, S. (2001). “Body schema and intentionality,” in The Body and The Self, 3rd Edn, eds J. L. Bermudez, A. J. Marcel, and N. M. Eilan (Cambridge: The MIT Press), 225–244.

Google Scholar

Gallagher, S. (2005). How the Body Shapes the Mind. New York, NY: Oxford University Press. doi: 10.1093/0199271941.001.0001

CrossRef Full Text | Google Scholar

Gallagher, S. (2012). “Why the body is not in the brain,” in Bodies in Action and Symbolic Forms: Two Sides of Embodiment Theory, Vol. 9, eds H. Bredekamp, M. Lauschke, and A. Arteaga (Berlin: Akademie Verlag), 273–288. doi: 10.1524/9783050095097.xxxi

CrossRef Full Text | Google Scholar

Gallagher, S. (2015). Invasion of the body snatchers: how embodied cognition is being disembodied. The Philosophers’. Magazine 68, 96–102. doi: 10.5840/tpm20156819

CrossRef Full Text | Google Scholar

Glenberg, A. M., and Gallese, V. (2012). Action-based language: a theory of language acquisition, comprehension, and production. Cortex 48, 905–922. doi: 10.1016/j.cortex.2011.04.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Glenberg, A. M., and Kaschak, M. P. (2002). Grounding language in action. Psychon. Bull. Rev. 9, 558–565. doi: 10.3758/BF03196313

CrossRef Full Text | Google Scholar

Goldman, A. I. (2012). A moderate approach to embodied cognitive science. Rev. Philos. Psychol. 3, 71–88. doi: 10.1007/s13164-012-0089-0

CrossRef Full Text | Google Scholar

Goldman, A. I., and de Vignemont, F. (2009). Is social cognition embodied? Trends Cogn. Sci. 13, 154–159. doi: 10.1016/j.tics.2009.01.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Head, H., and Holmes, G. (1911). Sensory disturbances from cerebral lesions. Brain 34, 102–254. doi: 10.1093/brain/34.2-3.102

CrossRef Full Text | Google Scholar

Holmes, N. P., and Spence, C. (2004). The body schema and multisensory representation (s) of peripersonal space. Cogn. Process. 5, 94–105. doi: 10.1007/s10339-004-0013-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Hommel, B. (1997). Toward an action-concept model of stimulus-response compatibility. Adv. Psychol. 118, 281–320. doi: 10.1016/S0166-4115(97)80041-6

CrossRef Full Text | Google Scholar

Hommel, B., Müsseler, J., Aschersleben, G., and Prinz, W. (2001). The Theory of Event Coding (TEC): a framework for perception and action planning. Behavioral and Brain Sciences 24, 849–937. doi: 10.1016/S0166-4115(97)80041-6

CrossRef Full Text

Hutto, D. D., and Myin, E. (2013). Radicalizing Enactivism: Basic minds without content. Cambridge/London: The MIT Press. doi: 10.1017/S0140525X01000103

CrossRef Full Text | Google Scholar

Hutto, D. D., and Myin, E. (2017). Evolving Enactivism. Basic Minds meet Content. Cambridge/London: The MIT Press.

Google Scholar

Keysers, C. (2009). Mirror Neurons. Curr. Biol. 19, R971–R973. doi: 10.1016/j.cub.2009.08.026

PubMed Abstract | CrossRef Full Text | Google Scholar

Keysers, C., Kaas, J. H., and Gazzola, V. (2010). Somatosensation in social perception. Nature reviews. Neuroscience 11, 417–428. doi: 10.1038/nrn2833

PubMed Abstract | CrossRef Full Text | Google Scholar

Keysers, C., Kohler, E., Umiltá, M. A., Nanetti, L., Fogassi, L., and Gallese, V. (2003). Audiovisual mirror neurons and action recognition. Exp. Brain Res. 153, 628–636. doi: 10.1007/s00221-003-1603-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Klee, P., and Glaesemer, J. (eds) (1979). Beiträge zur Bildnerischen Formlehre. Basel: Schwabe AG.

Klotz, W., and Neumann, O. (1999). Motor Activation Without Conscious Discrimination in Metacontrast Masking. Journal of Experimental Psychology: Human Perception and Performance 25, 976–992. doi: 10.1037/0096-1523.25.4.976

CrossRef Full Text | Google Scholar

Kornblum, S., Hasbroucq, T., and Osman, A. (1990). Dimensional overlap: Cognitive basis for stimulus–response compatibility—a model and taxonomy. Psychological Review 97, 253–270. doi: 10.1037/0033-295X.97.2.253

CrossRef Full Text | Google Scholar

Krois, J. M. (2011). “Bildkörper und Körperschema,” in John M. Krois. Bildkörper und Körperschema: Schriften zur Verkörperungstheorie ikonischer Formen, Vol. 2, eds H. Bredekamp and M. Lauschke (Berlin: Akademie Verlag), 252–271.

Lauschke, M. (2014). “Bodily resonance. Formative processes in aesthetic experience and developmental psychology,” in The Body in Relationship. Self, Other, Society, ed. C. Young (Stow: Body Psychotherapy Publications) 175–196.

Google Scholar

Lauschke, M. (2016). “Zur Interaktion mit Artefakten. Motorische Resonanz in Kunstpsychologie und Neurowisenschaft,” in Körper-Sprachen, eds S. Walz-Pawlita, B. Unruh, and B. Janta (Gießen: Psychosozial-Verlag), 36–57.

Lauschke, M. (2017a). “Dynamisierung von Bildräumen oder Resonanz als ästhetische Strategie gelingenden Lebens,” in Resonanz, Rhythmus und Synchronisierung. Erscheinungsformen und Effekte, eds T. Breyer, M. B. Buchholz, A. Hamburger, and S. Pfänder (Bielefeld: Transcript Verlag).

Lauschke, M. (2017b). “Motorische Resonanz,” in 23 Manifeste zu Bildakt und Verkörperung, eds M. Lauschke and P. Schneider (Berlin: Walter de Gruyter GmbH). doi: 10.1515/9783110561913

CrossRef Full Text | Google Scholar

Lipps, T. (1923). Ästhetik. Psychologie des Schönen und der Kunst. Erster Teil: Grundlegung der Ästhetik. Leipzig: Verlag von Leopold Voss.

Google Scholar

Louwerse, M. M. (2008). Embodied relations are encoded in language. Psychon. Bull. Rev. 15, 838–844. doi: 10.3758/PBR.15.4.838

CrossRef Full Text | Google Scholar

Makin, T. R., Holmes, N. P., and Ehrsson, H. H. (2008). On the other hand: dummy hands and peripersonal space. Behav. Brain Res. 191, 1–10. doi: 10.1016/j.bbr.2008.02.041

PubMed Abstract | CrossRef Full Text | Google Scholar

Makin, T. R., Holmes, N. P., and Zohary, E. (2007). Is that near my hand? Multisensory representation of peripersonal space in human intraparietal sulcus. J. Neurosci. 27, 731–740. doi: 10.1523/JNEUROSCI.3653-06.2007

PubMed Abstract | CrossRef Full Text | Google Scholar

Maravita, A., Spence, C., and Driver, J. (2003). Multisensory integration and the body schema: close to hand and within reach. Curr. Biol. 13, R531–R539. doi: 10.1016/S0960-9822(03)00449-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Müller-Freienfels, R. (1922). Psychologie der Kunst, 2nd edn. Leipzig: B. G. Teubner Verlag.

Google Scholar

Münsterberg, H. (1889). Beiträge zur experimentellen Psychologie, issue 1. Freiburg i. Br.: J.C.B. Mohr.

Neumann, O., Ansorge, U., and Klotz, W. (1998). Funktionsdifferenzierung im visuellen Kortex: Grundlage für motorische Aktivierung durch nicht bewußt wahrgenommene Reize? Psychol. Rundsch. 49, 185–196.

Oldfield, R. C. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97–113. doi: 10.1016/0028-3932(71)90067-4

CrossRef Full Text | Google Scholar

Prinz, W. (1990). “A common coding approach to perception and action,” in Relationships Between Perception and Action, eds O. Neumann and W. Prinz (Berlin: Springer-Verlag), 167–201. doi: 10.1007/978-3-642-75348-0

CrossRef Full Text | Google Scholar

Proverbio, A. M., Riva, F., and Zani, A. (2009). Observation of static pictures of dynamic actions enhances the activity of movement-related brain areas. PLoS One 4:e5389. doi: 10.1371/journal.pone.0005389

PubMed Abstract | CrossRef Full Text | Google Scholar

Pulvermüller, F. (2013). How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends Cog. Sci. 17, 458–470. doi: 10.1016/j.tics.2013.06.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Pulvermüller, F., Shtyrov, Y., Kujala, T., and Näätänen, R. (2004). Word-specific cortical activity as revealed by the mismatch negativity. Psychophysiology 41, 106–112. doi: 10.1111/j.1469-8986.2003.00135.x

PubMed Abstract | CrossRef Full Text | Google Scholar

R Development Core Team (2018). R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. Available at: https://www.R-project.org/

Google Scholar

Rizzolatti, G., and Craighero, L. (2004). The mirror-neuron system. Annu. Rev. Neurosci. 27, 169–192. doi: 10.1146/annurev.neuro.27.070203.144230

CrossRef Full Text | Google Scholar

Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. (1996). Premotor cortex and the recognition of motor actions. Cog. Brain Res. 3, 131–141. doi: 10.1016/0926-6410(95)00038-0

CrossRef Full Text | Google Scholar

Rosenbaum, D. A., and Kornblum, S. (1982). A priming method for investigating the selection of motor responses. Acta Psychol. 51, 223–243. doi: 10.1016/0001-6918(82)90036-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Rosenberg, R., and Klein, C. (2015). “The moving eye of the beholder. Eye-tracking and the perception of paintings,” in Art, Aesthetics, and The Brain, eds J. P. Huston, M. Nadal, F. Mora, L. F. Agnati, and C. J. Cela-Conde (Oxford: Oxford University Press), 79–108.

Google Scholar

Rueschemeyer, S. A., Pfeiffer, C., and Bekkering, H. (2010). Body schematics: on the role of the body schema in embodied lexical–semantic representations. Neuropsychologia 48, 774–781. doi: 10.1016/j.neuropsychologia.2009.09.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Sbriscia-Fioretti, B., Berchio, C., Freedberg, D., Gallese, V., and Umiltà, M. A. (2013). ERP modulation during observation of abstract paintings by Franz Kline. PloS One 8:e75241. doi: 10.1371/journal.pone.0075241

PubMed Abstract | CrossRef Full Text | Google Scholar

Schubert, T., Palazova, M., and Hutt, A. (2013). The time course of temporal attention effects on nonconscious prime processing. Atten. Percept. Psychophys. 75, 1667–1686. doi: 10.3758/s13414-013-0515-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Schütz-Bosbach, S., and Kuehn, E. (2014). “Experimentelle Handlungsforschung: die soziale Perspektive,” in Experimentelle Handlungsforschung, ed. W. Prinz (Stuttgart: Kohlhammer), 106–157.

Google Scholar

Shapiro, L. A. (2004). The Mind Incarnate. Cambridge: The MIT Press.

Google Scholar

Simon, J. R. (1990). The effects of an irrelevant directional cue on human information processing. Adv. Psychol. 65, 31–86. doi: 10.1016/S0166-4115(08)61218-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Sirigu, A., Duhamel, J.-R., and Poncet, M. (1991). The role of sensorimotor experience in object recognition: a case of multimodal agnosia. Brain 114, 2555–2573. doi: 10.1093/brain/114.6.2555

PubMed Abstract | CrossRef Full Text | Google Scholar

Tucker, M., and Ellis, R. (1998). On the relations between seen objects and components of potential actions. J. Exp. Psychol. Hum. Percept. Perform. 24:830. doi: 10.1037/0096-1523.24.3.830

CrossRef Full Text | Google Scholar

Uithol, S., van Rooij, I., Bekkering, H., and Haselager, P. (2011). Understanding motor resonance. Soc. Neurosci. 6, 388–397. doi: 10.1080/17470919.2011.559129

PubMed Abstract | CrossRef Full Text | Google Scholar

Umiltà, C., and Nicoletti, R. (1990). Spatial stimulus-response compatibility. Adv. Psychol. 65, 89–116. doi: 10.1016/S0166-4115(08)61219-4

CrossRef Full Text | Google Scholar

Umiltà, M. A., Berchio, C., Sestito, M., Freedberg, D., and Gallese, V. (2012). Abstract art and cortical motor activation: an EEG study. Front. Hum. Neurosci. 6:311. doi: 10.3389/fnhum.2012.00311

PubMed Abstract | CrossRef Full Text | Google Scholar

Vischer, R. (1873). Ueber das optische Formgefühl. Leipzig: Hermann Credner.

Wölfflin, H. (1886). Prolegomena zu einer Psychologie der Architektur. München: Kgl. Hof- und Universitäts-Buchdruckerei von Dr. C. Wolf & Sohn.

Google Scholar