Mathis Jording
Arne Hartz
Gary Bente
Martin Schulte-Rüther
Kai Vogeley- 1Department of Psychiatry and Psychotherapy, University Hospital Cologne, Cologne, Germany
- 2JARA-BRAIN, Aachen, Germany
- 3Translational Brain Research in Psychiatry and Neurology, Department of Child and Adolescent Psychiatry, Psychosomatics, and Psychotherapy, University Hospital RWTH Aachen, Aachen, Germany
- 4Department of Communication, Michigan State University, East Lansing, MI, United States
- 5Cognitive Neuroscience (INM-3), Institute of Neuroscience and Medicine, Research Center Jülich, Jülich, Germany
Humans substantially rely on non-verbal cues in their communication and interaction with others. The eyes represent a “simultaneous input-output device”: While we observe others and obtain information about their mental states (including feelings, thoughts, and intentions-to-act), our gaze simultaneously provides information about our own attention and inner experiences. This substantiates its pivotal role for the coordination of communication. The communicative and coordinative capacities – and their phylogenetic and ontogenetic impacts – become fully apparent in triadic interactions constituted in its simplest form by two persons and an object. Technological advances have sparked renewed interest in social gaze and provide new methodological approaches. Here we introduce the ‘Social Gaze Space’ as a new conceptual framework for the systematic study of gaze behavior during social information processing. It covers all possible categorical states, namely ‘partner-oriented,’ ‘object-oriented,’ ‘introspective,’ ‘initiating joint attention,’ and ‘responding joint attention.’ Different combinations of these states explain several interpersonal phenomena. We argue that this taxonomy distinguishes the most relevant interactional states along their distinctive features, and will showcase the implications for prominent social gaze phenomena. The taxonomy allows to identify research desiderates that have been neglected so far. We argue for a systematic investigation of these phenomena and discuss some related methodological issues.
Social Gaze as Special Case of Non-verbal Communication
Non-verbal communication does not only supplement verbal utterances but constitutes a crucial part of communication in itself. Thereby, non-verbal communication must not be treated as a series of isolated and discrete signals but as a complex and dynamic process (Burgoon et al., 1989, p. 23). In addition, the production and perception of non-verbal communication behavior are often implicit and automatic (Choi et al., 2005) – i.e., unintentional, uncontrollable processes humans are unaware of (Bargh, 1994).
Among the non-verbal cues, gaze behavior plays a pivotal role. The eyes are among the first and most frequently fixated regions in humans (Yarbus, 1967; Walker-Smith et al., 1977) from early infancy on (Haith et al., 1977), serve face and emotion recognition, and allow to identify gender, age, and personality (George and Conty, 2008; Itier and Batty, 2009).
The morphology of the human eye with its white sclera significantly enhances the visibility of the eyes and facilitates gaze recognition (Kobayashi and Kohshima, 1997, 2001), suggesting evolutionary adaptation to the increased importance of gaze-based social interaction and, eventually, social cognition in humans (Emery, 2000). Ontogenetically, attending to gaze can be considered a precursor of cooperation in young children (Tomasello et al., 2007). Both phylogenetically and ontogenetically (Grossmann, 2017) social gaze opens a “window into social cognition” (Shepherd, 2010).
In addition to coordination and management of verbal conversation (Argyle and Cook, 1976), gaze mutually coordinates attention which is a hallmark of social learning, communication, social interaction, and, finally, shared intentionality (Tomasello et al., 2007) and joint action (Sebanz and Knoblich, 2009). So-called joint attention (JA) is typically defined in the gaze domain: In triadic interactions (e.g., Lee et al., 1998), two persons can jointly attend to an object by one person following another person’s gaze toward a given object or possibly a third person. JA is the basis and prerequisite of cooperation (Tomasello et al., 2007) and has been investigated in great detail (Kleinke, 1986; Emery, 2000; Frischen et al., 2007; George and Conty, 2008; Itier and Batty, 2009; Shepherd, 2010; Falck-Ytter and von Hofsten, 2011; Pfeiffer et al., 2013b; Oberwelland et al., 2016; Grossmann, 2017).
The “Social Gaze Space” (SGS)
Despite the wealth of social gaze research, a unifying taxonomy of social gaze is still lacking. For the most commonly used taxonomy Emery (2000) summarized several core processes like averted gaze, mutual gaze, gaze following and JA under the term social gaze. However, this taxonomy has two major limitations: (1) the basic processes described by Emery were not considered as extended in time. Relatedly, transitions between states have not been taken into account. The taxonomy of Emery therefore lacks the complex and dynamic character of gaze encounters between two persons, which are extended in time and are based on the continuous exchange between the interactants. (2) An additional restriction of the traditional social gaze terminology and research is that they focus on explicit interactions in which at least one person deliberately tries to interact with or respond to another (Schilbach et al., 2010; Pfeiffer et al., 2014). However, already the mere presence of another person presumably strongly affects a persons’ behavior even when the partner is not interactively engaged. Recent research about the dual function of social gaze demonstrates that the awareness of someone else watching oneself can change the own gaze behavior (Gobel et al., 2015; Jarick and Kingstone, 2015). In accordance with recent interactionist advances emphasizing the dynamical character of interactions and arguing for ecological validity (Risko et al., 2012, 2016; Pfeiffer et al., 2013a; Schilbach et al., 2013), it is therefore important to consider all possible states of triadic interactions in a holistic approach.
In the following, we propose a taxonomy of the “Social Gaze Space” (SGS) that comprises all internal states a person can possibly adopt in the most basic setup of a gaze-based triadic interaction, as constituted by two interaction partners and an object1. These states are: partner-oriented (PO), object-oriented (OO), introspective (INT), responding joint attention (RJA), and initiating joint attention (IJA). We define these states on the basis of the behavior of one interactant (Figure 1). A dynamic interaction involving two persons can be conceptualized as a combination of two out of five different states which need not necessarily be temporally aligned. All combinations of states are possible and generate different types of interactional encounters that can be represented as a two-dimensional series of social gaze states evolving in time (Figure 2). This particularly applies to the interactive states of RJA and IJA, in which a person attempts to engage another person in an interaction which can be successful or not (see below section Triadic Interaction as a Dynamic Function of a Two-Dimensional State-Space). For this conceptualization, our focus lies on overt visual attention as deducible from gaze direction, whereas covert attention and other correlates of attention (e.g., pupil diameter, eye convergence, blinking rate) will be discussed only marginally.
FIGURE 1. Illustration of the five interactional states of the SGS (illustration in alignment with Emery, 2000) from the perspective of index person A (always the bold face at the bottom) in interaction with person B. (1) Partner-oriented: Attentional focus of A is directed toward B without deliberate attempts to interact of any of the two interactants. (2) Object-oriented: Attentional focus of A is directed toward an object within the shared environment. (3) Introspective: The attention is directed toward A’s own inner experience. (4) Responding JA: A follows B’s gaze toward an object. (5) Initiating JA: A tries to shift B’s attention toward an object.
FIGURE 2. All possible dual gaze states as a result of the combinations of gaze states of the two interactants (x- and y-axis). For illustration purposes states are presented in different order than previously introduced and as compared to Figure 1. Cell color indicates compatibility and stability of the states with white denoting compatible/stable states and gray denoting incompatible/unstable states. Red arrows suggest transitions from unstable to stable states. Green arrows exemplify the establishment of an interaction with a state of mutual interest serving as origin or gate (Note that arrows are not exhaustive of all possible transitions). The blue box (blue dashed line) designates states which methodologically can be inferred from a separate analysis of each participant. The purple box (purple dashed line) designates states which can only be inferred by an analysis of dynamics and interdependencies between the interactants.
The Five Gaze States
Partner-Oriented (PO)
In the partner-oriented state person A focuses her attention on person B. The eyes automatically attract visual attention (Laidlaw et al., 2012) and possibly convey information about personal attributes including gender, age and identity (Schyns et al., 2002), as well as emotional and attentional states (Baron-Cohen et al., 1997; Emery, 2000).
Eyes that focus on the viewer will be preferentially looked at (Senju and Hasegawa, 2005) or evaluated much more positively (Stass and Willis, 1967), modulate attention (Senju and Hasegawa, 2005; Dalmaso et al., 2017), increase emotional empathy (Schulte-Rüther et al., 2007) and modulate cognition suggesting a substantial ‘eye-contact-effect’ for diverse aspects of socio-emotional perception (Senju and Johnson, 2009). Among distractor stimuli, viewer-directed gaze is detected easily and much faster than averted gaze (von Grünau and Anston, 1995; Conty et al., 2006; Senju et al., 2008). Profound effects of viewer-direct eye gaze on preference (Hains and Muir, 1996) and attentional modulation (Farroni et al., 2002) have also been demonstrated in infants. This is probably the most thoroughly studied gaze state.
The effect of eye contact is much stronger during dynamic interactions with real persons than when confronted with static pictures (Hietanen et al., 2008; Pönkänen et al., 2011). This requires interactive approaches with dynamic face-to-face interactions (Pfeiffer et al., 2012, 2013a; Risko et al., 2012, 2016; Schilbach et al., 2013; Schilbach, 2014; Oberwelland et al., 2016).
Object-Oriented (OO)
In the object-oriented state person A’s attention is focused more or less entirely on an object in the shared environment, but not on the other person (as opposed to joint attention states described below during which person A oscillates between objects and person B). That is B’s presence and behavior are likely to influence A to some level but merely coincidentally and probably without A’s awareness. The exploration of different objects in a visual scenery is affected by the saliency of objects and thus the probability of persons directing their attention toward the objects (Itti and Koch, 2000). However, top-down as well as bottom-up processes are actively working together or compete for attention (Egeth and Yantis, 1997). Again, our attention and behavior toward objects are altered by actions or even the mere presence of another person looking at us (Senju and Hasegawa, 2005). Gaze cueing can automatically lead the attention toward particular objects (Frischen et al., 2007), even overriding the effect of higher psychophysical saliency (Borji et al., 2014). This brief instance of social interaction might induce a lasting attentional shift from a state of OO to the state of RJA [as examined in section Responding Joint Attention (RJA)]. However, even in the absence of any active gaze cuing, the presence of another person can attract covert attention (Kuhn et al., 2016; Laidlaw et al., 2016). Furthermore, the mere knowledge of the possibility of someone else watching their gaze lets participants control their gaze behavior with respect to its social adequacy (Risko and Kingstone, 2011).
Introspective (INT)
In this state person A neither focuses on objects nor on persons in the environment but only on his inner experience. Attentional disengagement from the outside world has been shown to correlate with a decrease in saccade frequency and an increase in saccade amplitude (Benedek et al., 2017) and, accordingly, a decrease in fixation frequency and an increase in fixation duration (Reichle et al., 2010; Benedek et al., 2017). Furthermore, in these situations blinking rate can increase (Smilek et al., 2010) and blinking duration can be prolonged (Salvi et al., 2015; Benedek et al., 2017). INT seems to show more variability in pupil diameter than episodes of directed attention to outward stimuli (Smallwood et al., 2011; Benedek et al., 2017). A higher variability of eye vergence (Benedek et al., 2017) suggests a less focused gaze (Solé Puig et al., 2013).
While it is intuitively obvious that these changes are indicative of a reduced responsiveness to events in the outside world (Smallwood et al., 2011; Benedek et al., 2017), it is an open question whether the reduced responsiveness to external stimuli and the overall change in gaze behavior are both the result and an epiphenomenon of INT, or whether changes such as a decrease in the frequency of microsaccades during INT may represent active visual disengagement as a strategy to achieve reduced responsiveness (Benedek et al., 2017). Another strategy participants adopt in situations of high cognitive load is to avoid looking into the eyes of an observer because this would entail higher demands on cognitive processing (Glenberg et al., 1998; Doherty-Sneddon and Phelps, 2005; Phelps et al., 2006; Markson and Paterson, 2009). Interestingly, the additional cognitive demands of mutual gaze do not seem to originate in the physical properties of the stimulus (e.g., the eyes) but in the interactive character inherent in this situation (Markson and Paterson, 2009). It is therefore crucial to consider introspective attentional states as potentially socially influenced by the presence of another person.
Responding Joint Attention (RJA)
In the responding JA state person A waits for B to initiate and lead the interaction, e.g., B chooses an object and A follows B’s gaze toward the object. Gaze following reactions that respond to the invitation of another person thereby establishing a rudimentary form of JA appear to be deeply rooted in human behavior (Pfeiffer et al., 2011). The gaze of another person automatically cues one’s own attention even when it is uninformative (Friesen and Kingstone, 1998), and participants exhibit gaze following even for forthright counter-predictive gaze cues (Driver et al., 1999; Bayliss and Tipper, 2006).
Gaze following with the aim of establishing JA constitutes a very simple though effective mechanism allowing for the inference of the attentional focus of other persons. The ability to adopt the attentional focus of another person is a prerequisite for reinforcement learning, from infants to adults (Vernetti et al., 2017). Infants at 6 months of age are already able to follow the eyes of other persons, in particular in a communicative context (Senju and Csibra, 2008). Accordingly, early proficiency in gaze following in infants predicts the development of mentalizing and emergence of language (Morales et al., 1998; Charman et al., 2000). JA and gaze following facilitate social learning, social competence, self-regulation, intelligence, and depth of information processing (Mundy and Newell, 2007).
Initiating Joint Attention (IJA)
In this state, person A takes the lead within the interaction by initiating JA. While gaze following in RJA reflects person A’s understanding that B’s perception and actions are goal-directed or have communicative intent, the initiation of JA is considered to require elaborate processing and insight (Tomasello and Carpenter, 2005). To initiate JA, A has to acknowledge (1) the dual function of social gaze (Gobel et al., 2015; Jarick and Kingstone, 2015) i.e., that gaze does not only serves her in perceiving but also that her gaze informs B about her focus of attention and, (2) sharing of attention is a desirable aim for mutual interaction (Tomasello et al., 2005). Whereas first elements of RJA are already evident at 6 months of age, IJA does not emerge before the second year of life (Mundy and Newell, 2007; Mundy et al., 2007). Chimpanzees followed the experimenters gaze on a frequent basis but did not try to initiate JA (Tomasello and Carpenter, 2005). Interestingly, differential development of both RJA and IJA can be observed in brain systems from childhood to adulthood (Oberwelland et al., 2016), as well as during atypical development in disorders such as autism (Oberwelland et al., 2017). In autism, IJA is typically more impaired than RJA and emerges much later than in typical development (Mundy, 2003). These empirical findings clearly point toward separate underlying cognitive systems of RJA and IJA (Mundy and Newell, 2007).
The innate tendency to expect other humans to follow their gaze (Pfeiffer et al., 2011) corresponds to the perception of successful initiation of JA as rewarding (Schilbach et al., 2010; Pfeiffer et al., 2014; Oberwelland et al., 2016). A successfully initiated instance of JA alters the consecutive interaction by increasing the tendency to look at and dwell upon the partners face (Bayliss et al., 2013).
Triadic Interaction as a Dynamic Function of a Two-Dimensional State-Space
Having defined the basic states during triadic JA, the picture becomes more complex when considering that each of the two participants can adhere to any of these states during a triadic interaction unfolding in time. In theory, a dual social state may be one of 25 possible combinations (representing varying degrees of “interactivity”), spanning a two dimensional SGS (Figure 2; see McCall and Singer, 2015 for an alternative concept of a 2D gaze space). Some of these combinations might be more ephemeral than others: e.g., a person A might soon lose the motivation to initiate JA if person B does not respond to him adequately, person A might switch to PO very soon subsequently (‘stability’ of states is indicated by cell color in Figure 2, with gray cells indicating unstable and ephemeral states; red arrows represent subsequent shifts from unstable to stable states).
Furthermore, it is conceivable that mutual attention (PO/PO) might facilitate transitions from non-interactive to interactive states (indicated by green arrows in Figure 2). These transitions have yet to be empirically investigated. Only non-interactive states (blue box in Figure 2) can be understood on the basis of single persons whereas the study of interactive situations (purple box in Figure 2) requires a complex dynamic concept and experimental setup, based on the idea that the basic unit of analysis is the interaction between both interactants.
Reflections and Future Directions
It is our goal to provide a unifying taxonomy of social gaze in triadic interactions and their respective interdependencies. This complex, dynamic and holistic approach has two major achievements. First, it facilitates the integration of existing empirical findings within one unifying framework and helps to identify research desiderates. Second, it will go beyond many of the previous studies that investigated gaze behavior in isolation and it will provide a theoretical background to study the complex dynamics of dual states including their transitions, thereby increasing the ecological validity of the empirical approaches.
This approach is in accordance with a growing number of proposals that argued in favor of “embedded” interactionist or “enactive” approaches and emphasize the importance of ecological validity in non-verbal communication and social cognition research (Kingstone, 2009; Marsh et al., 2009; De Jaegher et al., 2010; Konvalinka and Roepstorff, 2012; Risko et al., 2012, 2016; Skarratt et al., 2012; Gallagher, 2013; Pfeiffer et al., 2013b; Schilbach et al., 2013). New methodological approaches due to technological advances increasingly allow for the development of paradigms which meet those demands (Pfeiffer et al., 2013b; Oberwelland et al., 2016, 2017).
This paves the way to research questions concerning the nature of gaze communication in triadic interactions. Even in triadic encounters which are not explicitly interactive interactants are still likely to exert subtle influences on each other in many reciprocal ways: In PO, dynamic interactions elicit a much stronger eye contact effect that static pictures (Hietanen et al., 2008; Pönkänen et al., 2011); In OO, the visual attention of another person will influence object processing in an observer in multiple ways (Reid et al., 2004; Bayliss et al., 2006; Becchio et al., 2008); the oculomotor changes observable in INT might be an active form of visual disengagement (Benedek et al., 2017). Therefore, a separate examination of allegedly interactive and non-interactive states in triadic interactions is not adequate. From the new unifying perspective of the SGS the very first step must be to systematically describe and identify the characteristics of gaze behavior associated with the individual gaze states. However, given the dynamic and continuous nature of non-verbal communication (Burgoon et al., 1989) our appreciation of the interactants experience of the encounter relies on our comprehension of transitions between interactional states. The consequential next step will then be the identification of potentially complex signifiers of these transitions in gaze behavior, yet unknown (e.g., gaze patterns characteristic for active attempts to catch the partners attention to reach a full-fledged state of JA), which can serve as indicators of these transitions in future studies.
We speculate that transitions between gaze states of the individual interactants are not independent, but are contingent upon each other to a changing degree. If these contingencies are crucial in the establishment of states of higher interactivity and phenomena like synchrony and rapport between interactants, then it should be possible to establish their causal role in experimental paradigms. The dual state of mutual attention (PO/PO) as a candidate state for a gate to higher degrees of interactivity (Figure 2) – as soon as its role is empirically corroborated – could be a potential starting point in these investigations.
Having established the prototypical SGS it is worth studying individual differences in the behavior and experiences in triadic gaze interactions. Questions which to the best of our knowledge have not been tackled before concern the relationship between specific personality traits and gaze behavior in triadic encounters and to which degree personality traits are ascribed on the basis of gaze behavior. Other obvious topics relate to developmental factors in the SGS and how and when children access the SGS or the effect of impairments in non-verbal communication as observable in autism have in the SGS.
Author Contributions
All authors substantially contributed to the conception of the work. MJ drafted the manuscript. AH, GB, MS-R, and KV revised it critically.
Funding
This work is a part of the project ‘EMOTISK’ from the ‘Bringing technology to the people’ research program on human–machine interaction, funded by the German Federal Ministry of Education and Research (Grant Number for MS-R: 16SV7242, for KV: 16SV7244). MS-R received funding by the Excellence Initiative of the German federal and state governments.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The authors like to thank their respective work units for their continued support and constructive discussions. The authors want to express their gratitude to Joseph Walther from Nanyang Technological University for helpful and substantial feedback at the 3rd Annual International Summer School at Michigan State University and at the University Hospital Cologne.
Footnote
- ^Although, our taxonomy explicitly comprises a set of discrete states, we use the term “Social Gaze Space” throughout the manuscript instead of the more precise term “Social Gaze State Space,” for the sake of simplicity and comprehensibility.
References
Bargh, J. A. (1994). “The four horsemen of automaticity: awareness, intention, efficiency, and control in social cognition,” in Handbook of Social Cognition: Basic Processes; Applications, eds R. S. Wyer Jr. and T. Srull (Hillsdale, NJ: Lawrence Erlbaum), 1–40.
Baron-Cohen, S., Wheelwright, S., and Jolliffe, T. (1997). Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger Syndrome. Vis. Cogn. 4, 311–331. doi: 10.1080/713756761
Bayliss, A. P., Murphy, E., Naughtin, C. K., Kritikos, A., Schilbach, L., and Becker, S. I. (2013). “Gaze leading”: initiating simulated joint attention influences eye movements and choice behavior. J. Exp. Psychol. Gen. 142, 76–92. doi: 10.1037/a0029286
Bayliss, A. P., Paul, M. A., Cannon, P. R., and Tipper, S. P. (2006). Gaze cuing and affective judgments of objects: I like what you look at. Psychon. Bull. Rev. 13, 1061–1066. doi: 10.3758/BF03213926
Bayliss, A. P., and Tipper, S. P. (2006). Predictive gaze cues and personality judgments: should eye trust you? Psychol. Sci. 17, 514–520. doi: 10.1111/j.1467-9280.2006.01737.x
Becchio, C., Bertone, C., and Castiello, U. (2008). How the gaze of others influences object processing. Trends Cogn. Sci. 12, 254–258. doi: 10.1016/j.tics.2008.04.005
Benedek, M., Stoiser, R., Walcher, S., and Körner, C. (2017). Eye behavior associated with internally versus externally directed cognition. Front. Psychol. 8:1092. doi: 10.3389/fpsyg.2017.01092
Borji, A., Parks, D., and Itti, L. (2014). Complementary effects of gaze direction and early saliency in guiding fixations during free viewing. J. Vis. 14, 3–3. doi: 10.1167/14.13.3
Burgoon, J. K., Buller, D. B., and Woodall, W. G. (1989). Nonverbal Communication: The Unspoken Dialogue. Manhattan, NY: Harper & Row.x
Charman, T., Baron-Cohen, S., Swettenham, J., Baird, G., Cox, A., and Drew, A. (2000). Testing joint attention, imitation, and play as infancy precursors to language and theory of mind. Cogn. Dev. 15, 481–498. doi: 10.1016/S0885-2014(01)00037-5
Choi, Y. S., Gray, H. M., and Ambady, N. (2005). “The glimpsed world: Unintended communication and unintended perception,” in The New Unconscious, eds R. R. Hassin, J. S. Uleman, and J. A. Bargh (New York, NY: Oxford University Press), 309–333.
Conty, L., Tijus, C., Hugueville, L., Coelho, E., and George, N. (2006). Searching for asymmetries in the detection of gaze contact versus averted gaze under different head views: a behavioural study. Spat. Vis. 19, 529–545. doi: 10.1163/156856806779194026
Dalmaso, M., Castelli, L., Scatturin, P., and Galfano, G. (2017). Trajectories of social vision: eye contact increases saccadic curvature. Vis. Cogn. 25, 358–365. doi: 10.1080/13506285.2017.1290727
De Jaegher, H., Di Paolo, E., and Gallagher, S. (2010). Can social interaction constitute social cognition? Trends Cogn. Sci. 14, 441–447. doi: 10.1016/j.tics.2010.06.009
Doherty-Sneddon, G., and Phelps, F. G. (2005). Gaze aversion: a response to cognitive or social difficulty? Mem. Cognit. 33, 727–733. doi: 10.3758/BF03195338
Driver, J., Davis, G., Ricciardelli, P., Kidd, P., Maxwell, E., and Baron-Cohen, S. (1999). Gaze perception triggers reflexive visuospatial orienting. Vis. Cogn. 6, 509–540. doi: 10.1080/135062899394920
Egeth, H. E., and Yantis, S. (1997). Visual attention: control, representation, and time course. Annu. Rev. Psychol. 48, 269–297. doi: 10.1146/annurev.psych.48.1.269
Emery, N. J. (2000). The eyes have it: the neuroethology, function and evolution of social gaze. Neurosci. Biobehav. Rev. 24, 581–604. doi: 10.1016/S0149-7634(00)00025-7
Falck-Ytter, T., and von Hofsten, C. (2011). How special is social looking in ASD: a review. Prog. Brain Res. 189, 209–222. doi: 10.1016/B978-0-444-53884-0.00026-9
Farroni, T., Csibra, G., Simion, F., and Johnson, M. H. (2002). Eye contact detection in humans from birth. Proc. Natl. Acad. Sci. U.S.A. 99, 9602–9605. doi: 10.1073/pnas.152159999
Friesen, C. K., and Kingstone, A. (1998). The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychon. Bull. Rev. 5, 490–495. doi: 10.3758/BF03208827
Frischen, A., Bayliss, A. P., and Tipper, S. P. (2007). Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychol. Bull. 133, 694–724. doi: 10.1037/0033-2909.133.4.694
Gallagher, S. (2013). The socially extended mind. Cogn. Syst. Res. 2, 4–12. doi: 10.1016/j.cogsys.2013.03.008
George, N., and Conty, L. (2008). Facing the gaze of others. Neurophysiol. Clin. 38, 197–207. doi: 10.1016/j.neucli.2008.03.001
Glenberg, A. M., Schroeder, J. L., and Robertson, D. A. (1998). Averting the gaze disengages the environment and facilitates remembering. Mem. Cognit. 26, 651–658. doi: 10.3758/BF03211385
Gobel, M. S., Kim, H. S., and Richardson, D. C. (2015). The dual function of social gaze. Cognition 136, 359–364. doi: 10.1016/j.cognition.2014.11.040
Grossmann, T. (2017). The eyes as windows into other minds: an integrative perspective. Perspect. Psychol. Sci. 12, 107–121. doi: 10.1177/1745691616654457
Hains, S. M. J., and Muir, D. W. (1996). Infant sensitivity to adult eye direction. Child Dev. 67, 1940–1951. doi: 10.2307/1131602
Haith, M. M., Bergman, T., and Moore, M. J. (1977). Eye contact and face scanning in early infancy. Science 198, 853–855. doi: 10.1126/science.918670
Hietanen, J. K., Leppänen, J. M., Peltola, M. J., Linna-aho, K., and Ruuhiala, H. J. (2008). Seeing direct and averted gaze activates the approach–avoidance motivational brain systems. Neuropsychologia 46, 2423–2430. doi: 10.1016/j.neuropsychologia.2008.02.029
Itier, R. J., and Batty, M. (2009). Neural bases of eye and gaze processing: the core of social cognition. Neurosci. Biobehav. Rev. 33, 843–863. doi: 10.1016/j.neubiorev.2009.02.004
Itti, L., and Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Res. 40, 1489–1506. doi: 10.1016/S0042-6989(99)00163-7
Jarick, M., and Kingstone, A. (2015). The duality of gaze: eyes extract and signal social information during sustained cooperative and competitive dyadic gaze. Front. Psychol. 6:1423. doi: 10.3389/fpsyg.2015.01423
Kingstone, A. (2009). Taking a real look at social attention. Curr. Opin. Neurobiol. 19, 52–56. doi: 10.1016/j.conb.2009.05.004
Kleinke, C. L. (1986). Gaze and eye contact: a research review. Psychol. Bull. 100, 78–100. doi: 10.1037/0033-2909.100.1.78
Kobayashi, H., and Kohshima, S. (1997). Unique morphology of the human eye. Nature 387, 767–768. doi: 10.1038/42842
Kobayashi, H., and Kohshima, S. (2001). Unique morphology of the human eye and its adaptive meaning: comparative studies on external morphology of the primate eye. J. Hum. Evol. 40, 419–435. doi: 10.1006/jhev.2001.0468
Konvalinka, I., and Roepstorff, A. (2012). The two-brain approach: how can mutually interacting brains teach us something about social interaction? Front. Hum. Neurosci 6:215. doi: 10.3389/fnhum.2012.00215
Kuhn, G., Teszka, R., Tenaw, N., and Kingstone, A. (2016). Don’t be fooled! Attentional responses to social cues in a face-to-face and video magic trick reveals greater top-down control for overt than covert attention. Cognition 146, 136–142. doi: 10.1016/j.cognition.2015.08.005
Laidlaw, K. E. W., Risko, E. F., and Kingstone, A. (2012). A new look at social attention: orienting to the eyes is not (entirely) under volitional control. J. Exp. Psychol. Hum. Percept. Perform. 38, 1132–1143. doi: 10.1037/a0027075
Laidlaw, K. E. W., Rothwell, A., and Kingstone, A. (2016). Camouflaged attention: covert attention is critical to social communication in natural settings. Evol. Hum. Behav. 37, 449–455. doi: 10.1016/j.evolhumbehav.2016.04.004
Lee, K., Eskritt, M., Symons, L. A., and Muir, D. (1998). Children’s use of triadic eye gaze information for “mind reading.” Dev. Psychol. 34, 525–539.
Markson, L., and Paterson, K. B. (2009). Effects of gaze-aversion on visual-spatial imagination. Br. J. Psychol. 100, 553–563. doi: 10.1348/000712608X371762
Marsh, K. L., Richardson, M. J., and Schmidt, R. C. (2009). Social connection through joint action and interpersonal coordination. Top. Cogn. Sci. 1, 320–339. doi: 10.1111/j.1756-8765.2009.01022.x
McCall, C., and Singer, T. (2015). Facing off with unfair others: introducing proxemic imaging as an implicit measure of approach and avoidance during social interaction. PLoS One 10:e0117532. doi: 10.1371/journal.pone.0117532
Morales, M., Mundy, P., and Rojas, J. (1998). Following the direction of gaze and language development in 6-month-olds. Infant Behav. Dev. 21, 373–377. doi: 10.1016/S0163-6383(98)90014-5
Mundy, P. (2003). Annotation: the neural basis of social impairments in autism: the role of the dorsal medial-frontal cortex and anterior cingulate system. J. Child Psychol. Psychiatry 44, 793–809. doi: 10.1111/1469-7610.00165
Mundy, P., Block, J., Delgado, C., Pomares, Y., Van Hecke, A. V., and Parlade, M. V. (2007). Individual differences and the development of joint attention in infancy. Child Dev. 78, 938–954. doi: 10.1111/j.1467-8624.2007.01042.x
Mundy, P., and Newell, L. (2007). Attention, joint attention, and social cognition. Curr. Dir. Psychol. Sci. 16, 269–274. doi: 10.1111/j.1467-8721.2007.00518.x
Oberwelland, E., Schilbach, L., Barisic, I., Krall, S. C., Vogeley, K., Fink, G. R., et al. (2016). Look into my eyes: investigating joint attention using interactive eye-tracking and fMRI in a developmental sample. Neuroimage 130, 248–260. doi: 10.1016/j.neuroimage.2016.02.026
Oberwelland, E., Schilbach, L., Barisic, I., Krall, S. C., Vogeley, K., Fink, G. R., et al. (2017). Young adolescents with autism show abnormal joint attention network: a gaze contingent fMRI study. Neuroimage Clin. 14, 112–121. doi: 10.1016/j.nicl.2017.01.006
Pfeiffer, U. J., Schilbach, L., Jording, M., Timmermans, B., Bente, G., and Vogeley, K. (2012). Eyes on the mind: investigating the influence of gaze dynamics on the perception of others in real-time social interaction. Front. Psychol. 3:537. doi: 10.3389/fpsyg.2012.00537
Pfeiffer, U. J., Schilbach, L., Timmermans, B., Kuzmanovic, B., Georgescu, A. L., Bente, G., et al. (2014). Why we interact: on the functional role of the striatum in the subjective experience of social interaction. Neuroimage 101, 124–137. doi: 10.1016/j.neuroimage.2014.06.061
Pfeiffer, U. J., Timmermans, B., Bente, G., Vogeley, K., and Schilbach, L. (2011). A non-verbal turing test: differentiating mind from machine in gaze-based social interaction. PLoS One 6:e27591. doi: 10.1371/journal.pone.0027591
Pfeiffer, U. J., Timmermans, B., Vogeley, K., Frith, C. D., and Schilbach, L. (2013a). Towards a neuroscience of social interaction. Front. Hum. Neurosci 7:22. doi: 10.3389/fnhum.2013.00022
Pfeiffer, U. J., Vogeley, K., and Schilbach, L. (2013b). From gaze cueing to dual eye-tracking: novel approaches to investigate the neural correlates of gaze in social interaction. Neurosci. Biobehav. Rev. 37, 2516–2528. doi: 10.1016/j.neubiorev.2013.07.017
Phelps, F. G., Doherty-Sneddon, G., and Warnock, H. (2006). Helping children think: gaze aversion and teaching. Br. J. Dev. Psychol. 24, 577–588. doi: 10.1348/026151005X49872
Pönkänen, L. M., Alhoniemi, A., Leppänen, J. M., and Hietanen, J. K. (2011). Does it make a difference if I have an eye contact with you or with your picture? An ERP study. Soc. Cogn. Affect. Neurosci. 6, 486–494. doi: 10.1093/scan/nsq068
Reichle, E. D., Reineberg, A. E., and Schooler, J. W. (2010). Eye movements during mindless reading. Psychol. Sci. 21, 1300–1310. doi: 10.1177/0956797610378686
Reid, V. M., Striano, T., Kaufman, J., and Johnson, M. H. (2004). Eye gaze cueing facilitates neural processing of objects in 4-month-old infants. Neuroreport 15, 2553–2555. doi: 10.1097/00001756-200411150-00025
Risko, E. F., and Kingstone, A. (2011). Eyes wide shut: implied social presence, eye tracking and attention. Atten. Percept. Psychophys. 73, 291–296. doi: 10.3758/s13414-010-0042-1
Risko, E. F., Laidlaw, K., Freeth, M., Foulsham, T., and Kingstone, A. (2012). Social attention with real versus reel stimuli: toward an empirical approach to concerns about ecological validity. Front. Hum. Neurosci 6:143. doi: 10.3389/fnhum.2012.00143
Risko, E. F., Richardson, D. C., and Kingstone, A. (2016). Breaking the fourth wall of cognitive science: real-world social attention and the dual function of gaze. Curr. Dir. Psychol. Sci. 25, 70–74. doi: 10.1177/0963721415617806
Salvi, C., Bricolo, E., Franconeri, S. L., Kounios, J., and Beeman, M. (2015). Sudden insight is associated with shutting out visual inputs. Psychon. Bull. Rev. 22, 1814–1819. doi: 10.3758/s13423-015-0845-0
Schilbach, L. (2014). On the relationship of online and offline social cognition. Front. Hum. Neurosci. 8:278. doi: 10.3389/fnhum.2014.00278
Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., et al. (2013). Toward a second-person neuroscience. Behav. Brain Sci. 36, 393–414. doi: 10.1017/S0140525X12000660
Schilbach, L., Wilms, M., Eickhoff, S. B., Romanzetti, S., Tepest, R., Bente, G., et al. (2010). Minds made for sharing: initiating joint attention recruits reward-related neurocircuitry. J. Cogn. Neurosci. 22, 2702–2715. doi: 10.1162/jocn.2009.21401
Schulte-Rüther, M., Markowitsch, H. J., Fink, G. R., and Piefke, M. (2007). Mirror neuron and theory of mind mechanisms involved in face-to-face interactions: a functional magnetic resonance imaging approach to empathy. J. Cogn. Neurosci. 19, 1354–1372. doi: 10.1162/jocn.2007.19.8.1354
Schyns, P. G., Bonnar, L., and Gosselin, F. (2002). Show me the features! Understanding recognition from the use of visual information. Psychol. Sci. 13, 402–409. doi: 10.1111/1467-9280.00472
Sebanz, N., and Knoblich, G. (2009). Prediction in joint action: what, when, and where. Top. Cogn. Sci. 1, 353–367. doi: 10.1111/j.1756-8765.2009.01024.x
Senju, A., and Csibra, G. (2008). Gaze following in human infants depends on communicative signals. Curr. Biol. 18, 668–671. doi: 10.1016/j.cub.2008.03.059
Senju, A., and Hasegawa, T. (2005). Direct gaze captures visuospatial attention. Vis. Cogn. 12, 127–144. doi: 10.1080/13506280444000157
Senju, A., and Johnson, M. H. (2009). The eye contact effect: mechanisms and development. Trends Cogn. Sci. 13, 127–134. doi: 10.1016/j.tics.2008.11.009
Senju, A., Kikuchi, Y., Hasegawa, T., Tojo, Y., and Osanai, H. (2008). Is anyone looking at me? Direct gaze detection in children with and without autism. Brain Cogn. 67, 127–139. doi: 10.1016/j.bandc.2007.12.001
Shepherd, S. V. (2010). Following gaze: gaze-following behavior as a window into social cognition. Front. Integr. Neurosci. 4:5. doi: 10.3389/fnint.2010.00005
Skarratt, P. A., Cole, G. G., and Kuhn, G. (2012). Visual cognition during real social interaction. Front. Hum. Neurosci. 6:196. doi: 10.3389/fnhum.2012.00196
Smallwood, J., Brown, K. S., Tipper, C., Giesbrecht, B., Franklin, M. S., Mrazek, M. D., et al. (2011). Pupillometric evidence for the decoupling of attention from perceptual input during offline thought. PLoS One 6:e18298. doi: 10.1371/journal.pone.0018298
Smilek, D., Carriere, J. S. A., and Cheyne, J. A. (2010). Out of mind, out of sight: eye blinking as indicator and embodiment of mind wandering. Psychol. Sci. 21, 786–789. doi: 10.1177/0956797610368063
Solé Puig, M., Pérez Zapata, L., Aznar-Casanova, J. A., and Supèr, H. (2013). A role of eye vergence in covert attention. PLoS One 8:e52955. doi: 10.1371/journal.pone.0052955
Stass, J. W., and Willis, F. N. (1967). Eye contact, pupil dilation, and personal preference. Psychon. Sci. 7, 375–376. doi: 10.3758/BF03331131
Tomasello, M., and Carpenter, M. (2005). The emergence of social cognition in three young chimpanzees. Monogr. Soc. Res. Child Dev. 70, vii-132. doi: 10.1111/j.1540-5834.2005.00324.x
Tomasello, M., Carpenter, M., Call, J., Behne, T., and Moll, H. (2005). Understanding and sharing intentions: the origins of cultural cognition. Behav. Brain Sci. 28, 675–691. doi: 10.1017/S0140525X05000129
Tomasello, M., Hare, B., Lehmann, H., and Call, J. (2007). Reliance on head versus eyes in the gaze following of great apes and human infants: the cooperative eye hypothesis. J. Hum. Evol. 52, 314–320. doi: 10.1016/j.jhevol.2006.10.001
Vernetti, A., Smith, T. J., and Senju, A. (2017). Gaze-contingent reinforcement learning reveals incentive value of social signals in young children and adults. Proc. R. Soc. B Biol. Sci. 284: 20162747. doi: 10.1098/rspb.2016.2747
von Grünau, M., and Anston, C. (1995). The detection of gaze direction: a stare-in-the-crowd effect. Perception 24, 1297–1313. doi: 10.1068/p241297
Walker-Smith, G. J., Gale, A. G., and Findlay, J. M. (1977). Eye movement strategies involved in face perception. Perception 6, 313–326. doi: 10.1068/p060313
Keywords: non-verbal communication, social gaze, joint attention, triadic interaction, ecological validity, taxonomy, social psychology
Citation: Jording M, Hartz A, Bente G, Schulte-Rüther M and Vogeley K (2018) The “Social Gaze Space”: A Taxonomy for Gaze-Based Communication in Triadic Interactions. Front. Psychol. 9:226. doi: 10.3389/fpsyg.2018.00226
Received: 06 October 2017; Accepted: 09 February 2018;
Published: 26 February 2018.
Edited by:
Mario Weick, University of Kent, United KingdomReviewed by:
Matthias S. Gobel, University College London, United KingdomCade Andrew McCall, University of York, United Kingdom
Copyright © 2018 Jording, Hartz, Bente, Schulte-Rüther and Vogeley. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Mathis Jording, bWF0aGlzLmpvcmRpbmdAdWsta29lbG4uZGU=