Radosław Sterna
Agnieszka Małgorzata Siry
Joanna Pilarczyk1
Michał Jakub Kuniecki- 1Emotion and Perception Lab, Department of Philosophy, Institute of Psychology, Jagiellonian University, Krakow, Poland
- 2Department of Philosophy, Institute of Psychology, Jagiellonian University, Krakow, Poland
Introduction
Over the last 10 years, we have seen a growing interest in the usage of psychophysiology–i.e., examining physiological signals, such as electrodermal activity, heart rate, and electroencephalography, to study psychological phenomena–in Virtual Reality (VR) psychological research. The change is reflected in a rise in Google Scholar query results—from 811 in 2009 to 2,500 in 2019. This increase reveals a significant shift in the methodology of this field where both direct (e.g., questionnaires) and indirect measures of users' experience (e.g., psychophysiology) are used. At the same time, research interest in the social applications of VR technology (i.e., using VR to communicate, interact, and stay in presence of agents/avatars) has been consistently growing (discussion: Churchill et al., 2012).
The opinions on the usage of psychophysiology in VR research vary from enthusiastic (e.g., Blascovich et al., 2002; Meehan et al., 2002; Wiederhold and Rizzo, 2005; Bombari et al., 2015; Kisker et al., 2019) to reserved (e.g., Slater, 2004; Friedman et al., 2005; Llobera et al., 2010, p. 11). In this paper, we reflect on this polarization, discussing the possible advantages, limitations, and future directions of this methodology in the context of building impactful VR communication platforms.
Valuable Applications
First, we will discuss some valuable applications of psychophysiology that in our opinion can be beneficial for social VR, namely, (1) making VR research more objective, (2) enabling control over virtual character development, and (3) increasing engagement.
Making Research More Objective
From the early years of studying VR-mediated interactions, with both other users and bots, researchers aimed to quantify the VR experience. After the initial spread of the paper–pencil methods (e.g., Barfield and Hendrix, 1995; Witmer and Singer, 1998), more objective measurement (e.g., proximity measurement; Bailenson et al., 2004) was introduced. This trend was supported by Slater's (2004) heavy critique of the subjective measurement of presence, based on the inaccuracy of the declarative measurement on the one hand and the first attempts to use psychophysiological measurement in Virtual Environments (VE) on the other (Meehan et al., 2002; Hoyt et al., 2003). Recent studies show that psychophysiology may indeed be one of the ways of making VR-mediated interaction studies more objective by looking for physiological correlates of, e.g., engagement (Czarnek et al., 2020), social presence (Syrjämäki et al., 2020), or anxiety levels (Mühlberger et al., 2007; Crescentini et al., 2016).
Virtual Characters' Creation
Another promising direction is employment of the psychophysiological measures to test the virtual characters' creation, that is, a process of making decisions about its exact graphical form. In a virtual social situation, the interaction is mediated primarily by the spectator's visual perception of a virtual character. Therefore, we believe that it is essential to put more effort into the testing phase of virtual characters' creation, which would allow to standardize social virtual experience. For instance, in a recent study, Syrjämäki et al. (2020) investigated the effects of eye contact on social presence. We think that this is a particularly well-designed experiment as it combines precisely defined single experimental manipulation (eye-gaze) with subjective-free psychophysiological measurement (EDA). We find it promising to take this step-by-step, bottom-up approach in virtual character development that aims to break down complex stimuli into their basic elements. Ultimately, it can provide us with reliable knowledge of how to manipulate the virtual characters' features for making VR social interactions more engaging.
Engagement Increase
Usually, VR developers aim to create environments that engage their users in the virtual world and consequently facilitate meaningful interactions and authentic communication in VE. Some of the researchers try to make use of physiology for creating engaging training simulators (e.g., Czarnek et al., 2020; Muñoz et al., 2020) and games (Dynamic Difficulty Adjustment; e.g., Liu et al., 2009; Chanel and Lopes, 2020). Although not yet tested enough (Barreda-Ángeles et al., 2018), in future studies, the psychophysiological methodology might become a useful tool for studying engagement in VR social situations by being an indirect, unbiased indicator of the characteristics of interaction experience. A new line of research may be focused on transferring findings based on psychophysiology to creating engaging environments in the gaming or simulator training contexts or to creating meeting spaces where interactants could feel “as if” in real social situations.
Limitations
The social VR research is a relatively new field, at the crossroads between social and technical sciences suffering from common methodological problems characteristic for interdisciplinary research (Schmälzle and Meshi, 2020). Not all researchers share enthusiasm toward the use of indirect measurement in VR. Here, we discuss some of the limitations of this approach. In particular, we focus on (1) the danger of assuming isomorphic relationship between the signal and the psychological construct and the usage of (2) complex experimental design and (3) unstandardized stimuli.
Assuming an Isomorphic Relationship
First, a recurring problem is assuming an isomorphic (one-to-one) relationship between the physiological signal (e.g., heart rate) and the psychological construct (e.g., presence) (Cacioppo et al., 2007, p. 804). The limitations of causal relationship inference between physiology and psychological construct were raised by several researchers in the field (Llobera et al., 2010, p. 11; Kivikangas et al., 2011; general discussion: Cacioppo and Tassinary, 1990). As Cacioppo and Tassinary (1990, p. 24) underline “simply knowing that manipulating a particular psychological element [.] leads to a particular physiological response [.] does not logically enable inferences about the former based on observations of the latter.” Isomorphic assumption is made in many papers, both relatively old (Meehan et al., 2002) and recent (Hartanto et al., 2014, p. 2–10; Athif et al., 2020; Gill, 2020). This practice might create an illusion of having solid support for research findings when there is none. The VR research can be especially susceptible to that problem due to close ties between the research and industry. It might be tempting for VR researchers working under the market pressure to take this assumption to meet the expectations of the developers wanting their devices to be “objectively” validated.
Usage of Complex Design
The second limitation is the usage of complex design—introducing too many experimental manipulations to a single experiment. That, along with unstandardized stimuli employment (to be discussed below), may be a serious threat to the internal validity of the study (Stemmler, 2003). The more complex design, the higher the chance that change in physiological signal is due to confounders and not independent variables. Stemmler (2003, p. 241) calls physiological response “the slave of many masters,” meaning that the change in the physiological signal can be due to: the stimuli, the psychological processes, as well as the action performed by the participant. Therefore, it is important to limit the impact of the confounders by using rather simple designs preferably based on Mill (1862) method of difference. Unfortunately, in VR-based research, it is common to violate this method by creating conditions greatly differing in terms of the scenario, sensory input, virtual character characteristics, user's possible actions, etc. (e.g., Vinayagamoorthy et al., 2004, p. 149–152; Slater et al., 2006; Llobera et al., 2010, p. 4–7; Hartanto et al., 2014, p. 3–7). Consequently, it becomes impossible to causally connect the resulting physiological signal change with any particular part of experimental manipulation.
Lack of Pre-testing
Third, a specific example of the inability to control for confounders is the lack of pretesting of the stimuli and procedures used in the VR-based social experiments. As Emmerich and Masuch (2016) point out, lack of pretesting is one of the problems connected to the research in VR conducted with virtual characters (for discussion: Sterna et al., 2019). Besides virtual characters, experimental procedures in VR experiments are usually also not pretested, made ad hoc, without the usage of standardized materials or performing pilot tests (with few exceptions, e.g., Garau et al., 2005, p. 107; Zimmer et al., 2019, p. 7–12; Harjunen, 2019, p. 34–48; Niu et al., 2020, p. 060413-3). When not pretested, the efficacy of the experimental manipulation cannot be ensured. As a result, the studies become incomparable, and accumulation of knowledge impossible.
Future Directions
In the last part of the paper, we wish to reflect on the possible methodological directions of VR-mediated interactions research inspired by classical psychophysiology (Cacioppo et al., 2007). The solutions we propose aim to optimize the experimental design and results interpretability (Cacioppo et al., 2007, p. 848). We think that plenty of errors can be reduced by taking measures presented in the following sections. For analogous discussion regarding the methodological issues on the junction between communication science and neuroscience, see Schmälzle and Meshi (2020).
Higher Precision in Experimental Design
Firstly, we encourage the researchers to strive for higher precision in experimental design to reduce the impact of possible confounding factors. As aforementioned, one of the limitations of the current VR-based psychophysiological research is the usage of complex designs with many factors manipulated between the conditions. Following the current recommendations on psychophysiological research (Cacioppo et al., 2007), we suggest simplifying the manipulations made in the experiments by limiting them to one characteristic of environment or virtual character or other single feature at a time (e.g., Pan and Slater, 2007, p. 102–104; Harjunen, 2019, p. 34–38). Additionally, detailed descriptions of virtual character actions (as in, e.g., Pan et al., 2012) should be included. This approach might better warrant the precise identification of the cause of change observed in the physiological signal as a dependent variable.
Standardization and Pretesting
Secondly, we encourage the creation of standardized databases of experimental stimuli or at least pretesting the stimuli before the study (see current guidelines: Kourtesis et al., 2020). It is a common practice in psychophysiological experiments to make use of databases of experimental stimuli (e.g., Nencki Affective Picture System, NAPS; Marchewka et al., 2014), and it seems possible to create similar databases of virtual characters or its components (example of VR-video database: Li et al., 2017). VR allows for systematic changes in the environments and virtual characters (e.g., Hale et al., 2015) with precise control over all aspects of the presented stimuli (Wiederhold and Rizzo, 2005). Adopting this approach in combination with standardization increases the interpretability of research findings by linking them clearly to single experimental manipulation. It is especially relevant when making use of high-level constructs, such as presence, copresence, realism, and immersion, as these are already non-unitary and multifaceted in nature.
Joint Analyses
Thirdly, we recommend the wider use of joint analysis of two types of temporal data: VE events and physiological signal (Friedman et al., 2005). Automatic extraction of time-logged VE events can be combined with continuous measurement of physiological data to analyze the phasic component of the signal. The event-related analysis is a powerful and commonly used method in psychophysiological studies (Cacioppo et al., 2007). Nevertheless, it is regrettably rarely used solution in VR research (Liebold et al., 2017). In this type of analysis, we extract only those parts of the signal that correspond to a defined repetitive event (e.g., an action of a virtual character), average those signal fragments over repetitions of a given experimental condition and statistically test it in comparison with the means obtained for other experimental conditions (Kivikangas et al., 2011). This approach is suitable for both passive experimental designs, e.g., free-viewing task (e.g., Syrjämäki et al., 2020), and active ones, requiring participants to perform repetitive events, e.g., reacting to specifically designated stimuli (e.g., Meehan et al., 2002). As every single measurement has a random error involved, by averaging the measurements over repetitive trials, we are getting closer to the true value of the signal, thereby increasing signal-to-noise ratio. Therefore, this design might lead to significant noise reduction and ensure better confounders control.
Discussion
In summary, in this paper, we reflected on the current voices raised regarding the usage of psychophysiology in social VR research. We critically discussed the methodological issues and proposed future directions in research on virtual social spaces. Although in the field of social VR research presently it is impossible to infer high-level psychological constructs, such as presence, engagement, or satisfaction, based solely on physiology (Barreda-Ángeles et al., 2018), we believe that measuring physiological changes in well-controlled conditions will bring us closer to designing VE fitted to human communication needs.
Author Contributions
RS wrote the major part of the paper, contributed to the conception of the paper, and performed the article search. AS wrote the minor part of the paper and performed the article search. JP and MK wrote the minor part of the paper. All authors listed have made substantial intellectual contribution to the work, revised the manuscript, and read and approved the submitted version.
Funding
This work was financed from the budget for science in the years 2019-2023, as a research project the impact of virtually generated characters: human aspects of bots (project number: DI2018 015848) under the Diamond Grant program financed by the Ministry of Education and Science of Poland.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Athif, M., Rathnayake, B. L. K., Nagahapitiya, S. M. D. B. S., Samarasinghe, S. A. D. A. K., Samaratunga, P. S., Peiris, R. L., et al. (2020). Using biosignals for objective measurement of presence in virtual reality environments. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2020, 3035–3039. doi: 10.1109/EMBC44109.2020.9176022
Bailenson, J. N., Aharoni, E., Beall, A. C., Guadagno, R. E., Dimov, A., and Blascovich, J. (2004). “Comparing behavioral and self-report measures of embodied agents' social presence in immersive virtual environments,” in Proceedings of the 7th Annual International Workshop on Presence (Valencia).
Barfield, W., and Hendrix, C. (1995). The effect of update rate on the sense of presence within virtual environments. Virtual Real. 1, 3–15. doi: 10.1007/BF02009709
Barreda-Ángeles, M., Redondo-Tejedor, R., and Pereda-Baños, A. (2018). Psychophysiological methods for quality of experience research in virtual reality systems and applications. IEEE COMSOC MMTC Commun. Front. 4, 14–20.
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., and Bailenson, J. N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychol. Inq. 13, 103–124. doi: 10.1207/S15327965PLI1302_01
Bombari, D., Schmid Mast, M., Canadas, E., and Bachmann, M. (2015). Studying social interactions through immersive virtual environment technology: virtues, pitfalls, and future challenges. Front. Psychol. 6:869. doi: 10.3389/fpsyg.2015.00869
Cacioppo, J. T., and Tassinary, L. G. (1990). “Psychophysiology and psychophysiological inference,” in Principles of Psychophysiology: Physical, Social, and Inferential Elements, eds J. T. Cacioppo and L. G. Tassinary (Cambridge University Press), 3–33.
Cacioppo, J. T., Tassinary, L. G., and Berntson, G, (eds). (2007). Handbook of Psychophysiology. Cambridge university press. doi: 10.13140/2.1.2871.1369
Chanel, G., and Lopes, P. (2020). “User evaluation of affective dynamic difficulty adjustment based on physiological deep learning,” in International Conference on Human-Computer Interaction (Cham: Springer), 23. doi: 10.1007/978-3-030-50353-6_1
Churchill, E. F., Snowdon, D. N., and Munro, A. J, (eds). (2012). Collaborative Virtual Environments: Digital Places and Spaces for Interaction. Springer Science & Business Media.
Crescentini, C., Chittaro, L., Capurso, V., Sioni, R., and Fabbro, F. (2016). Psychological and physiological responses to stressful situations in immersive virtual reality: differences between users who practice mindfulness meditation and controls. Comput. Hum. Behav. 59, 304–316. doi: 10.1016/j.chb.2016.02.031
Czarnek, G., Strojny, P., Strojny, A., and Richter, M. (2020). Assessing engagement during rescue operation simulated in virtual reality: a psychophysiological study. Int. J. Hum. Comput. Interact. 36, 464–476. doi: 10.1080/10447318.2019.1655905
Emmerich, K., and Masuch, M. (2016). “The influence of virtual agents on player experience and performance,” in Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (New York, NY: Association for Computing Machinery), 10–21. doi: 10.1145/2967934.2968092
Friedman, D., Brogni, A., Antley, A., Guger, C., and Slater, M. (2005). “Sharing and analysing presence experiments data,” in Presence 2005: 8th Annual International Workshop on Presence (London: ISPR Presence), 111–118.
Garau, M., Slater, M., Pertaub, D. P., and Razzaque, S. (2005). The responses of people to virtual humans in an immersive virtual environment. Presence Teleoperat. Virtual Environ. 14, 104–116. doi: 10.1162/1054746053890242
Gill, E. (2020). A Psychophysiological Approach to Evaluating Participant Experience in Virtual Environments. [Dissertation], University of Michigan, Ann Arbor, MI, United States.
Hale, J., Pan, X., and Hamilton, A. F. D. C. (2015). “Using interactive virtual characters in social neuroscience,” in IEEE Virtual Reality (Arles), 189–190. doi: 10.1109/VR.2015.7223359
Harjunen, V. J. (2019). Perception of Facial Emotional Expressions and Touch in Virtual Face-to-Face Interaction. [Dissertation], University of Helsinki, Helsinki, Finland.
Hartanto, D., Kampmann, I. L., Morina, N., Emmelkamp, P. G., Neerincx, M. A., and Brinkman, W. P. (2014). Controlling social stress in virtual reality environments. PLoS ONE. 9:e92804. doi: 10.1371/journal.pone.0092804
Hoyt, C. L., Blascovich, J., and Swinth, K. R. (2003). Social inhibition in immersive virtual environments. Presence Teleoper. Virtual Environ. 12, 183–195. doi: 10.1162/105474603321640932
Kisker, J., Gruber, T., and Schöne, B. (2019). Behavioral realism and lifelike psychophysiological responses in virtual reality by the example of a height exposure. Psychol. Res. 85, 68–81. doi: 10.1007/s00426-019-01244-9
Kivikangas, J. M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., et al. (2011). A review of the use of psychophysiological methods in game research. J. Gam. Virtual Worlds 3, 181–199. doi: 10.1386/jgvw.3.3.181_1
Li, B. J., Bailenson, J. N., Pines, A., Greenleaf, W. J., and Williams, L. M. (2017). A public database of immersive VR videos with corresponding ratings of arousal, valence, and correlations between head movements and self report measures. Front. Psychol. 8:2116. doi: 10.3389/fpsyg.2017.02116
Liebold, B., Brill, M., Pietschmann, D., Schwab, F., and Ohler, P. (2017). Continuous measurement of breaks in presence: psychophysiology and orienting responses. Media Psychol. 20, 477–501. doi: 10.1080/15213269.2016.1206829
Liu, C., Agrawal, P., Sarkar, N., and Chen, S. (2009). Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. Int. J. Hum. Comput. Interact. 25, 506–529. doi: 10.1080/10447310902963944
Llobera, J., Spanlang, B., Ruffini, G., and Slater, M. (2010). Proxemics with multiple dynamic characters in an immersive virtual environment. ACM Trans. Appl. Percept. 8, 1–12. doi: 10.1145/1857893.1857896
Marchewka, A., Zurawski, Ł., Jednoróg, K., and Grabowska, A. (2014). The nencki affective picture system (NAPS): introduction to a novel, standardized, wide-range, high-quality, realistic picture database. Behav. Res. Methods 46, 596–610. doi: 10.3758/s13428-013-0379-1
Meehan, M., Insko, B., Whitton, M., and Brooks, F. P. Jr (2002). Physiological measures of presence in stressful virtual environments. ACM Trans. Graph. 21, 645–652. doi: 10.1145/566654.566630
Mill, J. S. (1862). A System of Logic Ratiocinative and Inductive, 5th Edn, Vol. 1. London: Parker, Son and Bourn.
Mühlberger, A., Bülthoff, H. H., Wiedemann, G., and Pauli, P. (2007). Virtual reality for the psychophysiological assessment of phobic fear: responses during virtual tunnel driving. Psychol. Assess. 19, 340–346. doi: 10.1037/1040-3590.19.3.340
Muñoz, J. E., Quintero, L., Stephens, C. L., and Pope, A. T. (2020). A Psychophysiological model of firearms training in police officers: a virtual reality experiment for biocybernetic adaptation. Front. Psychol. 11:683. doi: 10.3389/fpsyg.2020.00683
Niu, Y., Wang, D., Wang, Z., Sun, F., Yue, K., and Zheng, N. (2020). User experience evaluation in virtual reality based on subjective feelings and physiological signals. Electro. Imag. 2020, 60411–60413. doi: 10.2352/ISSN.2470-1173.2020.13.ERVR-382
Pan, X., Gillies, M., Barker, C., Clark, D. M., and Slater, M. (2012). Socially anxious and confident men interact with a forward virtual woman: an experimental study. PLoS ONE. 7:e32931. doi: 10.1371/journal.pone.0032931
Pan, X., and Slater, M. (2007). “A preliminary study of shy males interacting with a virtual female,” in Presence: The 10th Annual International Workshop on Presence (Barcelona).
Schmälzle, R., and Meshi, D. (2020). Communication neuroscience: theory, methodology and experimental approaches. Commun. Methods Meas. 14, 105–124. doi: 10.1080/19312458.2019.1708283
Slater, M. (2004). How colorful was your day? Why questionnaires cannot assess presence in virtual environments. Presence Teleoper. Virtual Environ. 13, 484–493. doi: 10.1162/1054746041944849
Slater, M., Guger, C., Edlinger, G., Leeb, R., Pfurtscheller, G., Antley, A., et al. (2006). Analysis of physiological responses to a social situation in an immersive virtual environment. Presence Teleoper. Virtual Environ. 15, 553–569. doi: 10.1162/pres.15.5.553
Stemmler, G. (2003). “Methodological considerations in the psychophysiological study of emotion,” in Handbook of Affective Sciences, Vol. 37, eds R. J. Davidson, H. H. Goldsmith, and K. R. Scherer (New York, NY: Oxford University Press), 225–255.
Sterna, R., Strojny, P., and Rebilas, K. (2019). Can virtual observers affect our behavior? Soc. Psychol. Bull. 14, 1–18. doi: 10.32872/spb.v14i3.30091
Syrjämäki, A. H., Isokoski, P., Surakka, V., Pasanen, T. P., and Hietanen, J. K. (2020). Eye contact in virtual reality–a psychophysiological study. Comput. Hum. Behav. 112:106454. doi: 10.1016/j.chb.2020.106454
Vinayagamoorthy, V., Brogni, A., Gillies, M., Slater, M., and Steed, A. (2004). “An investigation of presence response across variations in visual realism,” in The 7th Annual International Presence Workshop (Valencia), 148–155.
Wiederhold, B. K., and Rizzo, A. (2005). Virtual reality and applied psychophysiology. Appl. Psychophysiol. Biofeedb. 30, 183–185. doi: 10.1007/s10484-005-6375-1
Witmer, B. G., and Singer, M. J. (1998). Measuring presence in virtual environments: a presence questionnaire. Presence 7, 225–240. doi: 10.1162/105474698565686
Keywords: psychophysiology, physiology, VR, VE, virtual
Citation: Sterna R, Siry AM, Pilarczyk J and Kuniecki MJ (2021) Psychophysiology in Studying VR-Mediated Interactions: Panacea or a Trick? Valuable Applications, Limitations, and Future Directions. Front. Virtual Real. 2:648318. doi: 10.3389/frvir.2021.648318
Received: 31 December 2020; Accepted: 01 March 2021;
Published: 29 April 2021.
Edited by:
Marko Orel, Prague University of Economics and Business, CzechiaReviewed by:
Erwan David, Goethe University Frankfurt, GermanyCopyright © 2021 Sterna, Siry, Pilarczyk and Kuniecki. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Radosław Sterna, cmFkb3NsYXdzdGVybmEmI3gwMDA0MDtnbWFpbC5jb20=