Maria Mikhailenko
Nadezhda Maksimenko
Mikhail Kurushkin- Chemistry Neuroeducation Laboratory, SCAMT Institute, ITMO University, Saint Petersburg, Russia
The concept of using eye-tracking in virtual reality for education has been researched in various fields over the past years. With this review, we aim to discuss the recent advancements and applications in this area, explain the technological aspects, highlight the advantages of this approach and inspire interest in the field. Eye-tracking has already been used in science for many decades and now has been substantially reinforced by the addition of virtual and augmented reality technologies. The first part of the review is a general overview of eye-tracking concepts, technical parts, and their applications. In the second part, the focus shifted toward the application of eye-tracking in virtual reality. The third part, first the description of the recently emerged concept of eye-tracking in virtual reality is given, followed by the current applications to education and studying, which has not been thoroughly described before. We describe the main findings, technological aspects, and advantages of this approach.
Introduction
Eye-tracking is a technology that has for a long time been used in the humanitarian, language (Kaushanskaya and Marian, 2007; Suvorov, 2015; Aryadoust and Ang, 2019), medical research (Pernice et al., 2014; Rigby et al., 2020), and quite recently it has found its way into the field of user experience design. One of the most well-known and cited examples of the use of eye-tracking is given in a study from Nielsen (2006), where people were browsing the internet and looking at the search engine results, while their gaze was tracked. It has been shown that their attention is scattered on the page in the shape of a triangle or letter F. This research has then been followed up and proved later in other studies (Brunyé et al., 2019). Active use of eye-tracking technology over recent years has facilitated the delivery of information to the consumer through the media. Eye-tracking has been also applied to biomedical research and showed good results in the rehabilitation measures after ischemic brain damage (Krupinski et al., 2006; Cameirão et al., 2016; Faria et al., 2016; Peshkovskaya et al., 2017; Maggio et al., 2019a,c,d).
The successful use of eye-tracking technology in medical treatment has influenced the research to turn toward the use of eye-tracking with healthy patients. Thus, when studying the decision-making process of an individual related to the need for social interactions in a virtual environment, it was demonstrated that eye-tracking technologies can be used as a forecasting tool. The frequency and duration of fixation, the peak velocity, and amplitude of saccades can be used to identify and eliminate transient states of uncertainty when making perceptual decisions (Maggio et al., 2019b), as well as to assess the complexity of a problem when it becomes too complicated due to many details (Brunyé and Gardony, 2017). In predicting decision making, gaze fixation, fixation duration, and the number and duration of visits are of great importance. Sustainable combinations of these parameters form various starring strategies (Spinks and Mortimer, 2016).
Moreover, with the help of eye-tracking, it is possible to identify higher nervous functions such as emotions, regret, and disappointment (Nakhaeizadeh et al., 2020), as well as memorization processes (Bault et al., 2016). It has also been shown that eye-tracking technologies can be used to control decision-making (Helbing et al., 2020) for example by increasing its value (Fridman et al., 2018). A recent study (Smith and Krajbich, 2019) showed the importance of the multiplicative model, which states that greater attention to options when making a choice, has higher influences on choices. Another important study has demonstrated (Pärnamets et al., 2015) that the number of eye movements between informationally important objects in the context of the current task can be used as an indicator, which has an inverse relationship to the quality of solving a cognitive task.
Research in the field of eye movements and eye trajectories analyses in the context of virtual three-dimensional space are fairly new. Some works are aimed at studying the emotional behavior of a person interacting with objects in a virtual space, in particular studies (Rosa et al., 2017; Reichenberger et al., 2020), implicit eye movements and their significance were analyzed with regards to the influence of attention on the emotional learning processes. Some researchers are studying the possibility of forming interactive interfaces for influencing the virtual environment by changing the trajectory of the gaze (Paletta et al., 2020). Research in the field of education is also beginning to appear, for example, in Piotrowski and Nowosielski (2020), the possibility of constructing a predictive system for students’ skill level was studied by analyzing their trajectory of gaze in a virtual environment. Part of the research is also aimed at analyzing the usability of virtual reality (VR) interfaces by studying the gaze trajectory fixed in virtual three-dimensional space (Orlosky et al., 2019). Thus, over the past 5 years, sufficient experience has been gained in using eye-tracking as a technology for predicting, evaluating, and managing the solution of emotional and cognitive tasks.
The concept and applications of eye-tracking in education has already been described in several review papers. Rappa et al. (2019) have investigated different ways in which eye tracking can be implemented into the learning process by providing a detailed scoping review with measures and emerging patterns of eye tracking in learning environments. A recent review on applications of eye tracking into the digital learning environment (Gorbunovs, 2021) summarized the concept of eye tracking and current challenges in the field that have to be considered when applying eye tracking into the learning environment, such as the cultural background, gender, age, and literacy of the prospective users. Another new interesting work published in the special issue dedicated to the Research of Visual Perception in Educational Science (Kaakinen, 2021) indicates issues of the eye tracking research related to the operationalization of the theoretical constructs. More specifically, the paper explores the basic concepts of eye tracking, the measure of students’ engagement as well as the measure of teachers’ expertise. Moreover, the paper goes on to explore eye-movement as an indicator of student-teacher interaction. The second major point raised in the paper, is the analyses of eye tracking data.
Although there has been extensive research and many review papers on the topic of eye-tracking technology in various education areas, the concept of eye-tracking in virtual reality for educational purposes is a rather new direction that has only appeared in the last few years. Compared to previous review papers (Lai et al., 2013; Brunyé et al., 2014; Tien et al., 2014; Asan and Yang, 2015; Alemdag and Cagiltay, 2018; Weiss et al., 2018; Aryadoust and Ang, 2019; Rappa et al., 2019; Radianti et al., 2020), the current review combines both eye-tracking descriptions in various fields, focusing on education, and virtual reality, as well as their combination, which allows a novel perception of the existing techniques and technologies aimed at the improvement of the education process. With the existing knowledge, it is important to highlight the current progress in the area of eye-tracking in virtual reality for educational purposes. The study focuses on the existing applications in various fields of education as well as on proposing new ones. Over the past years, the most reported area of education with regards to eye tracking is medical education, many examples of which are mentioned in the current review. However, one of the aims of the article is to highlight the importance of research and implementation of technology in other areas of natural sciences education.
The article consists of three parts: a brief overview of eye-tracking techniques and application fields with the highlights of major developments, the implementation of eye-tracking in virtual reality in different fields of science, and the description of ongoing new research in the field of eye-tracking in virtual reality and education.
Literature Review
For the current review, the Scopus Electronic Library had been used to conduct the literature analysis. Various search queries were used, which included combinations of keywords in the Title, Abstract, and Keywords fields. For searching the articles that are related to various aspects of eye-tracking technology these combinations of keywords have been used:
“eye tracking,” “eye movement,” “eye tracking” AND “behavioral science,” “eye tracking” AND “education,” “eye tracking” AND “medicine,” “eye tracking” AND “design,” “eye tracking” AND “virtual reality,” “eye tracking” AND “virtual reality” AND “education,” “eye tracking” AND “virtual reality” AND “studying,” “eye tracking” AND “virtual reality” AND “teaching,” “eye tracking” AND “virtual reality” AND “learning.”
For the 4th section of the current review, the initial search query included “eye tracking” AND “virtual reality” and resulted in a total of 1,632 papers. Out of them, 644 were obtained by limiting the presence of the word “learning.” There were 409–64% articles of Computer Science subject area, 204–32% of Engineering, 129–20% of Medicine, 105–16% of Mathematics, and 99–15% of Neuroscience. Further filtering of the 1,632 articles by the word “training” gave 527 documents, where the ratio of subject areas was: Computer Science – 302 (55%) articles, Engineering – 179 (32%), Medicine – 138 (24%), Mathematics – 82 (14%), and Neuroscience – 62 (10%). The first search that included the word “teaching” resulted in 124 documents within the following subject areas: 62–48% articles of Computer Science, 38–29% of Medicine, 37–28% of Engineering, 26–19% of Social Sciences, and 17–12% of Mathematics. Also, 93 articles were discovered by including the word “studying” into the search query within these 1,632 documents: 64% of them were dedicated to Computer Science (61 articles), 21% to Engineering (21 articles), 18% to Medicine (19 articles), 15% to Neuroscience (16 articles), and 10% to Mathematics (11 articles). By including the word “education” into the search within initial 1,632 articles, 307 papers had been identified. The priority areas covered by these documents were: Computer Science (59% – 188 articles), Engineering (30% – 97 articles), Medicine (23% – 78 articles), Mathematics (13% – 47 articles), and Social Sciences (11% – 40 articles). A more detailed overview of the papers used in the current review, organized by sections of the review, is given in Table 1.
Table 1. Overview of the articles used for each section of the review by topic, virtual reality (VR) content, educational content, and educational method.
Eye-Tracking Concepts and Applications
When it comes to human cognition, eye-tracking is a tool that has extensively been used for various research (Rappa et al., 2019). The first eye tracker device came about in the early 20th century, and it consisted of special contact lenses that were used with a pointer attached to them. This technique was then changed and optimized by using light beams and recording their reflection on film instead (Huey, 1968; Duchowski, 2003). In the 1960’s modern eye-tracking approach had been developed, since then it has been further studied and improved (Buswell, 1937; Tatler et al., 2010). It is safe to say that the methods used for detecting eye movements have become significantly more accurate in recent years. At the moment, the most used methods involve video systems with computer vision techniques usage (Yarbus, 1967). Recent technological advances such as high-quality cameras have facilitated the use of eye-tracking devices of small size. These small devices can now be combined with portable “smart” glasses or a VR headset to provide accurate measures of eye movement and large data sets. Eye-tracking methods have been described in a variety of different experimental works (Hansen and Ji, 2010). Due to technological advances, eye-tracking is now widely and efficiently used in human cognition research.
Technological Aspects
The modern eye tracking devices can track the gaze by using an array of infrared or near-infrared light sources and cameras (Deubel and Schneider, 1996). The concept behind the majority of currently used system is based on an array of light sources illuminating the eye, which then produces a corneal reflection. The relationship between the produced reflection and the center of the eye pupil is then recorded, so that the vectors linking the eye position to the location in the perceived world can be calculated (Yarbus, 1967). The calculated viewpoint in space moves with the movement of eyes. Several hardware configurations of eye trackers are available on the market at the moment. These include systems that provide head stabilization, the so called «chin up» systems, as well as the remote ones that can only account for a limited amount of head movement. Moreover, there are also mobile eye-wear-based systems, which are the most modern of all. All of these hardware types have different advantages and disadvantages that are related to spatial (i.e., tracking) accuracy, tracking speed, mobility, portability, availability, and cost (Holmqvist et al., 2012; Funke et al., 2016).
Several indicators of visual behavior play an important role in understanding the interpretive processes behind gaze. Such indicators are, for example, position, motion, numeracy, and latency indicators (Deubel and Schneider, 1996). The fixations are the instantaneous pauses of the eyes that are measured in milliseconds. Saccades are the movements of the eye between successive fixations (Deubel and Schneider, 1996). Since the eye is constantly moving between fixation points, eye trackers are able to collect large amounts of data. Motion measurements provide quantitative analyses of the eye movement patterns in space during saccades. Examples of such measurements include the distance between successive saccades and saccade speed. Cartesian coordinate space is used to measure position of the gaze. The frequency of fixations and saccades is identified by the numerical measures of the eyes during scene perception. It is for example recorded, how these numerical values changes the position of the subject changes. The latency measures are then used to estimate the temporal dynamics of fixations and saccades, the example of these measures can include the duration of fixations and of saccades.
Once the fixations and sequential saccades have been recorded, the analysis is then carried out. The factors that are taken into consideration when analyzing are the internal and external states. The first includes states like frustration, uncertainty, anxiety, etc., while the latter is, for example, the organization of the stimulus. After explaining the technical side of the eye-tracking process, it is important to consider the areas of science and research where this technology has been most successfully applied in recent years.
Eye-Tracking in Behavioral Science
One of the fields where eye-tracking has been extensively used and studied over the last decade is without a doubt behavioral and cognitive science. There have been many studies that present eye-trackers for tracking changes in cognition (Lai et al., 2013; Holmqvist and Andersson, 2017). In one such study, it has been shown that the total viewing time and fixation time in areas corresponding to non-cooperative solutions are associated with the general level of participants’ cooperation. The increase in the total viewing time and fixation time on the areas that correspond to non-cooperative solutions happens due to the preference for non-cooperation by participants and a decrease in the overall level of cooperation. Therefore, the viewing time clearly correlates with the decision-making process. The number of fixations on group attributes is associated with group identity, but it does not immediately lead to cooperative behavior (Maggio et al., 2019d).
Eye-tracking technology has been a useful tool in research that is related to social attention. For example, it has been shown that results from certain experimental studies correlate with measures of social impairment and with autism symptom severity. It has been found that reduced attention to social stimuli or increased attention to non-social stimuli is correlated with behavioral measures of autism (Bird et al., 2011; Chawarska et al., 2012; Eckstein et al., 2017). Face processing, as well as language skills, are also significantly correlated with measures of social attention. A strong association between face processing skills and attention to faces has been reported for children (Shic et al., 2011), more studies suggested that attention to a speaker’s mouth and eyes could be a predictive measure of how fast the words are recognized among children with ASD (Parish-Morris et al., 2013). Moreover, when the processing of such social information as eye and mouth movement is atypical, it is correlated with difficulties in language learning or social impairment. An increase in attention toward the mouth has been associated with increased social adaptation (Tenenbaum et al., 2014) and communicative competence (Klin et al., 2002). These results suggest that eye-tracking methods are promising for studying social attention in ASD and can be successfully used in studies with children.
Eye-Tracking in Education
A major application and focus of this review is the use of eye-tracking in education (Norbury et al., 2009; Jarodzka et al., 2017, 2021; Strohmaier et al., 2020). Over the past decades, there has been substantial research on the fundamental concepts and applications of eye tracking in education. There are numerous great review papers on the topic some of which will be mentioned in this subsection.
In the work by Sun et al. (2018), it is argued that the application of the newest information technologies to traditional teaching methods can not only increase the value of technologies but also improve the progress in learning and effectively integrate various areas of education. A special educational software package has been developed based on eye-tracking technology. By analyzing data on the movement of students’ eyes, teachers can improve the quality of teaching by adapting the teaching structure, while students can focus more on their interests and develop personalized education strategies. The study, Lai et al. (2013) aims to show how eye-tracking technology has been applied to learning research. A total of 81 articles, including 113 studies, were selected from the Social Sciences Citation Index database from 2000 to 2012. The tendencies of eye-tracking technology in the educational process are studied under several specific topics, e.g., patterns of information processing and patterns of decision-making. This study concludes that the eye-tracking method provides a promising channel for researchers to link learning outcomes with cognitive processes.
A work by Alemdag and Cagiltay (2018) is a systematic review that is dedicated to the cognitive processes in multimedia learning studying with relevant variables through eye-tracking technology. In the review, 52 articles with 58 studies were analyzed. The results showed that there is a growing interest in the use of eye-tracking technology in multimedia learning research. Eye movement measurements allowed the two authors to conclude the cognitive processes of choice, organization, and integration. Also, one of the results was that the multimedia content itself, individual differences, and emotions were potential factors that could affect eye movement measurements. Eye-tracking has proved itself to be an important tool that provides specific spatial and temporal measures to monitor, measure, analyze, and evaluate educational processes (McNamara and Jain, 2019). The topic of eye tracking in education is further developed with specific application to medicine as the next subsection.
Eye Tracking in Medicine
Eye-tracking has for a number of years been widely implemented in medical research, which has been reported and reviewed in many papers over the past decade (Krupinski, 2005; Krupinski et al., 2013; Bond et al., 2014; Tien et al., 2014; Asan and Yang, 2015; Rigby et al., 2020). For example, eye-trackers have been used alongside machine learning to improve the diagnostics, and also to predict diagnostics errors before they can even occur. With the use of eye trackers, automatic cueing or feedback can be provided to learners during image examination (Voisin et al., 2013). Such automatic feedback is realized by parsing medical images into diagnostically relevant and non-relevant regions (ROIs), while using expert annotation or automated machine vision techniques (Brunyé et al., 2014; Mercan et al., 2016; Nagarkar et al., 2016). Once these regions are known to the system that is used for eye tracking, fixations on the important regions are recorded. The recordings are then used to study the spatial distribution of attention over a digital image and the time of fixations. Once there is enough data available, it can be fed into the machine learning algorithm to provide automated diagnostics. Another example of the use of eye-tracking is demonstrated in the work by Tien et al. (2014), the review includes studies describing the use of eye-tracking when performing, learning, or evaluating completion of a task or acquiring skills for surgeons. The reviewed literature demonstrates the ability of vision tracking to provide reliable quantitative data as an objective assessment tool with potential application in surgical training to improve performance. Vision tracking remains a promising area of research with the possibility of further implementation in the assessment of surgical skills.
Another significant area of medical research that is associated with the use of eye-tracking devices is related to rehabilitation. It has been shown that the combination of virtual reality technology and eye-tracking, improves rehabilitation of such brain functions as attention, memory, motor and visual-spatial abilities, and speech when compared to traditional therapy methods (Rigby et al., 2020). Moreover, it has been reported that virtual reality training was able to stimulate the patients’ motivation. The universal ability, which is not tied to the nature and specifics of the disease, to increase motivation and demonstrate better results in the rehabilitation of lost cognitive functions has been reported. This universal property was demonstrated for brain injuries (Barlinn et al., 2016) and neurodegenerative diseases (Krupinski et al., 2006; Maggio et al., 2019a,c). This finding allowed to reduce the total hospitalization time of patients while increasing the duration of rehabilitation training (Maggio et al., 2019a). These few examples demonstrate the importance of this technology in the medical field and the applications to medical education will be discussed further.
Research in the field of medical education suggests that the pattern of eye movement of students changes with regard to diagnostics as they progress in their studies. More specifically, the eye movements become more rapid and move toward the importance for diagnostic regions (Jarodzka et al., 2010). Hence, the eye movement of students becomes more and more like that of an expert, this process can be time and analyzed. It has also been suggested that this process can be accelerated by showing students video material of the expert’s eye movement, this method is called eye-movement modeling examples (EMMEs) (Frank and Danoff, 2007).
The technology and application of EMMEs has been rapidly developing and many interesting studies have already been produced. EMMEs include video of expert eye movement as well as audio description of the action by the expert (van Gog et al., 2009; Jarodzka et al., 2013). The scientific basis behind this method lies in neuroscience, where it has been shown that the brain can mirror actions, when another person’s action is observed. Such a response is known as a “mirror system” and can be integrated into the learning process (Calvo-Merino et al., 2005, 2006). This technology provides students with the unique opportunity to learn from the experts in the field without them being physically available, this brings the home- or distant education to a new level of quality, which is particularly relevant during the pandemic outbreak (Jarodzka et al., 2010). EMMEs method had been used outside the medical education field, one of the studies reported that the use of EMMEs improved the ability of notice aircraft inspectors to detect aircraft faults (Sadasivan et al., 2005). The same has been observed for circuit board inspectors (Stein and Brennan, 2004; Nalanagula et al., 2006). Moreover, EMMEs accelerated the speed of debugging for the software engineers (Mason et al., 2015). Reports show that with the use of EMMEs students can become better readers and solve logical tasks such as puzzles faster (Velichkovsky, 1995).
When the gaze is analyzed with the use of eye-trackers, in the medical diagnostic field it is seen that a sequence of saccades and fixation over a medical image appears. In one study, radiographers viewed the eye movements of either fellow novices or experts during the learning process and then interpreted a chest X-ray (Litchfield et al., 2010). Interestingly, it has been shown that the ability to notice and locate pulmonary conditions has improved in comparison to “free-search” not only from observing the experts but also from observing novices.
In a recent study, medical students observed a video of child epilepsy cases. The video was played in three different settings, in the first one the expert was narrating the video with voice, this was a control video sample. In the second, the eye-movement of the expert was traced onto the video by a small circle on the images. In the third video, the eye movements of the expert were also presented, but the area of the image, which the expert didn’t focus on, was blurred out. The results of this study showed an increase in the diagnostic of students after viewing the third video and no change for the first or the second. Therefore, specific viewing conditions can facilitate the use of a “mirror” system in the brain and enhance the learning process (Jarodzka et al., 2012).
Alongside applications in medical education and research, eye tracking and VR can also play a part in assessing the standards of teaching in the medical field. New frameworks for those studying and working in healthcare are being developed by the international accreditation establishment to ensure the highest level of professionals. An example of such a framework is Competency-based medical education (CBME), which is aimed at ensuring that healthcare workers have high expertise that isn’t just certified on paper but also proven by practice (Aggarwal and Darzi, 2006). In recent years CBME is already being implemented by including new standards of teaching, assessment, and curriculum into medical education and practice (Simpson et al., 2002; Frank and Danoff, 2007; Swing, 2007; Nasca et al., 2012; Holmboe et al., 2016). Due to these changes, we need new technology to facilitate the assessment procedure and eye-tracking in VR can surely become one of the major tools in this process.
Eye Tracking in Design
Variety of studies reported the use of eye tracking in design, for example in graphic design and other computer-based visual evaluation methods, which are used to measure the distribution of visual attention (Poole and Ball, 2005; Hollander et al., 2019). In a study from 2016, eye-tracking was used to assess computer-based visual tools, which help decision-making to investigate ecosystem services (Klein et al., 2016). It has been suggested that the ability of users to understand ecosystem services can be improved through the use of such decision support systems. Other studies have analyzed the how people can perceive landscapes, using the data from eye trackers such as fixation time, saccade amplitude, etc. (De Lucio et al., 1996; Dupont et al., 2013; Potocka, 2013). The opportunities, as well as challenges for the use of eye-tracking in the fields of cartography, geographic information science, spatial cognitions, etc., have recently been reported in a broad review (Kiefer et al., 2017). Eye-tracking has also been investigated in the built environment, for example, a study of a contextual guidance model with a Bayesian framework was used to predict regions of gaze fixation of people while they search for objects in space (e.g., pedestrian paintings on the street) (Torralba et al., 2006). Similar work has been done by Ehinger et al. (2009) with an addition of the pedestrians being present or absent in the picture. Eye-tracking in the built environment has also been used to find out about visual preferences that people have with regard to the general objects in public spaces (Noland et al., 2017). The effectiveness of GIS software was studied with eye-tracking when the subjects performed orientation tasks with the help of GIS (Kudelka and Dobesova, 2015). This study provided a useful insight into the way people were orientated, and the subjects that were successful in the self-localization task were found to spend more visual attention on objects that provided helpful clues. Recently, real-time eye-tracking systems have been implemented to increase the efficiency and quality of interactive graphics applications as well as large-scale display systems (Cheng and Pulo, 2003; Bergeron et al., 2015; Saxena et al., 2016; Celine et al., 2018; Weiss et al., 2018; Merali et al., 2019; Pottle, 2019; Rutkowski et al., 2019; Ustun et al., 2020).
Eye-Tracking in Virtual Reality
The natural development of technology and research in the field of eye-tracking had led to the combination of eye tracking with a virtual reality tool (McNamara and Jain, 2019). VR is a powerful tool that can change the way we work and relax; it can also transform learning techniques in the near future. New applications of VR are invented every day, so it is safe to say that it could be present in our daily life in a relatively short time. The extent of VR research is rather large and recently many virtual reality systems with eye-tracking have emerged. A vast part of VR research is aimed at improving user experience and reducing usability issues, and it is believed that eye-tracking technology can be of assistance in this task (Clay et al., 2019).
Essentially, eye-tracking gives the ability to identify what the user’s gaze is focusing on in the virtual reality environment (VRe). Moreover, VR can be used to change the focus of attention if it will increase the positive outcome of the task. There are techniques to draw attention to certain things in the VRe, which can be used if needed. The success of these techniques can be constantly checked since the gaze of the user is tracked in real-time. The use of eye-tracking in VR can be a helpful asset, not only improving the work of various applications but also identifying the disadvantages of some VRe. In VR, full-body motion tracking can be used, so the environment can react to the user’s movement, action, and gaze.
In comparison to real-world eye-tracking, the one in VR has the advantage of the easier definition of the regions that the user had looked at. These regions of interest can also be identified in time and reconstructed. The experimental setup with eye-tracking in virtual reality is much more flexible and promising for many fields of science since it can be thoroughly controlled. One can control the data collection, environmental settings, and make the stimuli more natural for the user, therefore enhancing the possibilities of research. It is especially useful in research that focuses on human cognition and behavior. It has been reported that such eye-movement signals as pupil diameter can be used for emotional recognition, and therefore there is a correlation between various emotions and pupil size (Hess and Polt, 1960). Research has provided a comparison between eye-tracking in VRe emotional recognition and the classic EEG approach. This research suggests that although the eye-tracking classification method was not as efficient as EEG, it still had statistically good enough results to be considered a useful tool for this task (Zheng et al., 2020).
Various implementations of VR and eye-tracking have been numerously reported in the field of computer science. Some research includes very detailed technical reports of VR and eye-tracking implementations. For example, a review paper by Clay et al. (2019) gives a step-by-step explanation of the available hardware and techniques used to implement eye tracking into the VR set. Other studies have been dedicated to the development of VR devices and the improvement of existing hardware. There is a particularly interesting study, which assesses what technical requirements a VR set needs to have in order to generate a well-known immersive effect of being “in reality,” or as researchers refer to it, in “presence” (Radianti et al., 2020).
A lot of recent VR research is being conducted in the area of computer gaming. The ability to use eye-gaze in the play has for a long time been a question of investigation and a desired feature for the gaming industry. This topic was given further rise with the launch of VR headsets with built-in eye-tracking, such as Vive, FOVE, and other devices available on the consumer market (Fove Inc, 2017; High Tech Computer Corporation HTC Vive Pro Eye, 2019). Apart from using gaze as an instrument of gameplay, eye-tracking can also collect data on where the attention of the gamer is scattered in 3D VRe, therefore, an adaptive game mechanism can create reactive content in the gameplay. One of the papers has shown that the use of gaze to interact with remote objects in the VRe is much faster than the use of hands (Tanriverdi and Jacob, 2000), which could be used as a great advantage for shooting or racing games or any other applications where special attention is paid to speed and aiming quality.
Gaze can also be used as means of non-verbal communication, especially useful in collaborative VR environments. There has been research showing that the use of VR can regulate the interaction between people in this process (Gergle et al., 2013). The integration of eye movement in avatar interactions is actively researched with the use of eye-tracking (Hußmann and Oechsner, 2017).
In the area of medical research, VR technology has been studied and integrated into the educational process. An example of this would be the study of nurse education in a collaborative immersive system (Weiss et al., 2018), or another research on the medical training in a virtual hospital and medical professional training (Saxena et al., 2016; Pottle, 2019). VR technology has been reported and researched in dental medicine, such as simulated cavities removal exercise for dental students in VR (Celine et al., 2018) or a surgical education system where finger tracking is used to show the students the location of the fingers and the exact movements of the expert’s fingers during surgery (Merali et al., 2019). Another use of VR in medicine is rehabilitation, where the technology is used on patients. Many papers regarding rehabilitation have been reported so far, an example is a VR-based therapy for vestibular problems (Bergeron et al., 2015), VR breathing exercises for people with Chronic Obstructive Pulmonary Disease (Rutkowski et al., 2019). VR has been used to visualize the body and actually move through the neural tissue; this feature has been used in medical research (Ustun et al., 2020). In recent years VR with eye-tracking has been appearing in the medical field. It has proven itself especially useful in the area of medical education and will be described below.
Here we have described the major areas of eye-tracking in VR research which has been rapidly developing in the last years. With the development of this scientific area, more and more research emerge on the advantages, and importance of VR in educational programs. This will be discussed in more detail in the next section.
Eye-Tracking in Virtual Reality for Education
A concept that has been the major interest of this review is eye-tracking in VR, which can be used in education to enhance the learning process and to assess the knowledge of students. This idea has been reported in some literature over the past decade, especially with regards to medical education, however, the field is still evolving and the wide use in the other fields is yet to be seen (Aggarwal and Darzi, 2006). Here, we describe the findings of the aforementioned papers and suggest ways in which the technology can be applied to the assessment of knowledge in the classroom. An immersive education environment has the functionality to carry out assessment procedures with minimal distraction. The knowledge level of the taken course as well as individual aspects of the student, such as cognitive abilities, and achievements can be taken into account. Therefore the learning trajectories of the students can be personalized to enable the best possible result. A particular planned study with a focus on eye-tracking in a virtual learning environment is described in a paper by Schlechtinger (2020), where the author argues that the literature has provided enough initial evidence for the use of eye-tracking in the context of learning. This planned study will evaluate how well eye-tracking works when it comes to detecting objects that cause excessive cognitive load in virtual reality conditions.
Some research has recently been conducted in the area of knowledge assessments, for example, students were asked to read a text or a fragment of text being in the virtual reality environment (Pilgrim and Pilgrim, 2016). It has been stated that their knowledge in a particular field can be judged by data collected from eye-trackers, more specifically, the concentration on specific words and expressions during the task (Mason et al., 2015).
While eye-tracking has already been used for years in medical education and training, it is argued that eye-tracking in VR will provide even more opportunities to enhance the learning process. One of the main findings in this field shows that there is a difference in the way experts and novices move their eyes (Jarodzka et al., 2010). It has been mentioned that the use of eye-tracking in medical education can perhaps decrease the time it takes a novice to become an expert by accessing gaze patterns.
The available research suggests that vision tracking in desktop VR in LCD display can be a useful tool for developing diagnostic skills in medical students (Litchfield et al., 2010). It is well known that the process of medical education and training is a long multi-step process, therefore, using eye-tracking in VR alongside conventional teaching techniques might be able to increase the efficiency and accelerate the training.
With the successful and effective implementation of eye-tracking in virtual reality into medical education and training, it makes sense to interpolate these findings into other fields of education. There is also the potential of implementing eye-tracking technologies in virtual reality to engineering education (Khokhar et al., 2019). Kang et al. (2020) describe the so-called Deepwater Horizon operation, where VR allows the assessment of the awareness of trainees and operators in a non-dangerous environment. One of the unobtrusive and viable methods for assessing situation awareness can be the use of eye movements, in particular, time-ordered visual scanning trajectories. Based on the presented data, it can be argued that a field of science education, which usually involves a lot of lab work, would greatly benefit from the novelty of eye-tracking in virtual reality. The personalization of learning trajectories in chemistry education can be achieved in a variety of ways.
The implementation of virtual reality can create an environment where students learn each topic at their own pace. For example, a recent technology report has shown how chemistry education can be enhanced with the use of VR (Maksimenko et al., 2021). Furthermore, eye-tracking could be used to collect data regarding the progress of a student on a chemical topic (Muna and Bahit, 2020; Vandenplas et al., 2021). It can be anticipated that the better knowledge students have, the less time they will spend looking at incorrect answers and the quicker they will concentrate their gaze on the correct one in tests. The same assessment can be done with regards to chemical exercises. Naturally, some students will learn faster than others, and eye-tracking could be a good measure of this process. The educational process can then be personalized, more on this will be specified in the outlook. The attention of the students can be measured as described by Yoshimura et al. (2019) where the study shows the development of attention-restoring visual signals for displaying when gaze tracking detects that students’ attention is shifting from important objects since in educational virtual reality it is important to solve the problems of students’ inattention to the presented content (Yoshimura et al., 2019). Experiments to compare different signals and their parameters to assess the effectiveness and trade-offs, as well as to assess the impact of eye tracking are proposed. Eye-tracking is used both to detect inattention and to control the appearance and location of signals.
Outlook for Eye-Tracking in Virtual Reality for Education
As eye trackers are becoming cheaper, more portable, easier to use, and generally more available to the consumers, research on principled methods for using eye-tracking for competency assessment is expected to increase (Al-Moteri et al., 2017). However, the implementation of eye-tracking in VR into the educational setting is a new concept of educational science, therefore, there are many directions for future research that we would like to point out.
First of all, more experiments in the educational environment are to be conducted with the use of eye-tracking in VR, not only in the aforementioned areas of medical education, but also other natural sciences. A major advantage of VR in the educational field is the ability to combine classic laboratory experience while eliminating dangers that are associated with it for school and university students. Therefore, the experimental work is necessary in order to establish the extent to which the learning process is inhibited in such an environment.
The wider use of the technology will provide more data sets that are crucial for understanding the effectiveness of such educational approaches. Moreover, large data sets can be analyzed to optimize the experience. They are also needed in order to establish a stronger correlation between cognition and education, and therefore ensure qualitative assessment for students when using eye-tracking. Thus, analyzing the data from the eye-trackers is another vital direction of the research.
However, the methods to be used when analyzing the data are not established. Thus, a major research direction is the development of techniques to evaluate the effectiveness of eye-tracking in education. This should be done by using both quantitative and qualitative research methods and assessing the knowledge of students, increased level of skill, overall learning experience as well as the downsides of the approach. For example, a possible method for assessment of engagement can be done by the implementation of a machine-based algorithm, as suggested for eye-tracking in education (Goldberg et al., 2021).
Eye-tracking can facilitate the transition to personalized education, where eye-trackers in VR education applications are to measure cognitive load, fatigue, tiredness, and concentration level of individual students. The development of methods that will be used in order to effectively convert data from eye-tracking devices into the quantitative assessment of students’ involvement is to be a major part of the research. Further research can potentially show how personalized education will effect cheating in tests and exams while in the classroom, as well as anxiety and peer pressure that motivate students to avoid extra questions on the topic.
It is important to investigate the drawbacks and limitations of the approach with respect to education but also the technical drawbacks. For example, one drawback can be that eye-trackers need to be individually calibrated, which is tedious and time-consuming in an experimental lab environment. Another is the movement of the headset which would lead to inaccurate data produced by the eye-tracker. Therefore, the position of the headset needs to be fixed before the activity, sharp and fast movements need to be avoided. Some of the problems related to physical discomfort can be solved by making a series of short experiments and having periods of rest or providing students with an active and engaging task to distract from physical discomfort. However, further research needs to be conducted with respect to the aforementioned limitations, motion sickness, and others that arise in the process of technology implementation.
Another fascinating research direction is the use of a multi-method approach, where eye-tracking can be combined with others by which eye-tracking is combined with other physical and cognitive measures in order to provide a more detailed insight into the learning process. An example of such a measure can be a head pose (Ballenghein et al., 2020; Goldberg et al., 2021).
It has already been described that eye-tracking in VR can be used to also test the expertise of teachers (Seidel et al., 2021) or increase it, as well as to assess student-teacher engagement (Haataja et al., 2021). We suggest that more detailed research should be done in these directions.
Together with many exciting ideas and advances, eye-tracking in VR also has a fair number of drawbacks. However, we suggest that exploring these technologies alongside each other, will facilitate and perhaps accelerate the overcoming of these problems. And inevitably it will bring these technological advances to many classrooms across the globe.
Conclusion
In conclusion, the current review has provided firstly an overview of the areas where eye-tracking technology has been researched and implemented over the past decades. A specific focus has been dedicated to eye-tracking in education and the areas of application. Secondly, it shows that VR technology has been widely applied in many fields of scientific research over the past years and the interest in it continues to grow. It can be observed that both these technologies are sophisticated and well researched, moreover, we argue that now they can be used together and more extensively applied to the area of education.
Implementation of eye-tracking in VR provides new interesting approaches for studying the attention and motivation of students, possibly accelerating and making the education more efficient as well as a tool for assessment. The ability to use VR with different environments, model and control every aspect of the process, makes it an indispensable educational tool. In this review, we provide a possible trajectory for the development and application of these technologies in the classroom.
It is worth noting that the development of new methods for using eye-tracking in VR for education is especially relevant in the current situation, with many countries worldwide switching to distant teaching and the educational system trying to adapt to the changes. The methods of eye-tracking in VR implementation with regards to hardware and software have been numerously reported, therefore this technology can soon become available in day-to-day life.
However, there are certain disadvantages to this technology, a major one of which being motion sickness and visual discomfort which appears while using VR for a longer time. It can prevent students from using the technology to its full potential or even using it at all. Nonetheless, these obstacles will be surpassed with the help of further research in the area.
Overall, we believe that eye-tracking in combination with VR presents a powerful tool that can change the way we perceive education and greatly expand the potential of modern educational programs. This combination has already been successfully applied in the field of medical education, therefore we suggest that more people from different fields should pay attention to this technology and consider the possibility of its implementation into the educational process.
Author Contributions
MK, NM, and MM conducted conceptualization and contributed to writing—review and editing. MK contributed to project administration and validation. MM contributed to writing—original draft. MM and NM contributed to later drafts of the manuscript.
Funding
This work was supported by the Russian Science Foundation, grant no. 20-78-10135.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Aggarwal, R., and Darzi, A. (2006). Technical-skills training in the 21st century. N. Engl. J. Med. 355, 2695–2696. doi: 10.1056/NEJMe068179
Alemdag, E., and Cagiltay, K. (2018). A systematic review of eye tracking research on multimedia learning. Comput. Educ. 125, 413–428. doi: 10.1016/j.compedu.2018.06.023
Al-Moteri, M. O., Symmons, M., Plummer, V., and Cooper, S. (2017). Eye tracking to investigate cue processing in medical decision-making: a scoping review. Comput. Human Behav. 66, 52–66. doi: 10.1016/j.chb.2016.09.022
Aryadoust, V., and Ang, B. H. (2019). Exploring the frontiers of eye tracking research in language studies: a novel co-citation scientometric review. Comput. Assist. Lang. Learn. 34, 898–933. doi: 10.1080/09588221.2019.1647251
Asan, O., and Yang, Y. (2015). Using eye trackers for usability evaluation of health information technology: a systematic literature review. JMIR Hum. Factors 2:e5. doi: 10.2196/humanfactors.4062
Ballenghein, U., Kaakinen, J. K., Tissier, G., and Baccino, T. (2020). Cognitive engagement during reading on digital tablet: evidence from concurrent recordings of postural and eye movements. Q. J. Exp. Psychol. 73, 1820–1829. doi: 10.1177/1747021820931830
Barlinn, J., Kozak, K., Kunka, B., and Kosikowski, R. (2016). “Brain rehabilitation in clinical trials setup by eye-tracking,” in Proceedings of the Fifth International Conference on Telecommunications and Remote Sensing, Milan, 89–94. doi: 10.5220/0006227500890094
Bault, N., Wydoodt, P., and Coricelli, G. (2016). Different attentional patterns for regret and disappointment: an eye-tracking study. J. Behav. Decis. Mak. 29, 194–205. doi: 10.1002/bdm.1938
Bergeron, M., Lortie, C. L., and Guitton, M. J. (2015). Use of virtual reality tools for vestibular disorders rehabilitation: a comprehensive analysis. Adv. Med. 2015:916735. doi: 10.1155/2015/916735
Bird, G., Press, C., and Richardson, D. C. (2011). The role of alexithymia in reduced eye-fixation in autism spectrum conditions. J. Autism Dev. Disord. 41, 1556–1564. doi: 10.1007/s10803-011-1183-3
Bond, R. R., Zhu, T., Finlay, D. D., Drew, B., Kligfield, P. D., Guldenring, D., et al. (2014). Assessing computerized eye tracking technology for gaining insight into expert interpretation of the 12-lead electrocardiogram: an objective quantitative approach. J. Electrocardiol. 47, 895–906. doi: 10.1016/j.jelectrocard.2014.07.011
Brunyé, T. T., Carney, P. A., Allison, K. H., Shapiro, L. G., Weaver, D. L., and Elmore, J. G. (2014). Eye movements as an index of pathologist visual expertise: a pilot study. PLoS One 9:e103447. doi: 10.1371/journal.pone.0103447
Brunyé, T. T., Drew, T., Weaver, D. L., and Elmore, J. G. (2019). A review of eye tracking for understanding and improving diagnostic interpretation. Cogn. Res. Princ. Implic. 4:7. doi: 10.1186/s41235-019-0159-2
Brunyé, T. T., and Gardony, A. L. (2017). Eye tracking measures of uncertainty during perceptual decision making. Int. J. Psychophysiol. 120, 60–68. doi: 10.1016/j.ijpsycho.2017.07.008
Calvo-Merino, B., Glaser, D. E., Grèzes, J., Passingham, R. E., and Haggard, P. (2005). Action observation and acquired motor skills: an fMRI study with expert dancers. Cereb. Cortex 15, 1243–1249. doi: 10.1093/cercor/bhi007
Calvo-Merino, B., Grèzes, J., Glaser, D. E., Passingham, R. E., and Haggard, P. (2006). Seeing or doing? Influence of visual and motor familiarity in action observation. Curr. Biol. 16, 1905–1910. doi: 10.1016/j.cub.2006.07.065
Cameirão, M. S., Faria, A. L., Paulino, T., Alves, J., and BermúdezBadia, S. (2016). The impact of positive, negative and neutral stimuli in a virtual reality cognitive motor rehabilitation task: a pilot study with stroke patients. J. Neuroeng. Rehabil. 13:70. doi: 10.1186/s12984-016-0175-0
Celine, G., Cho, V., Kogan, A., Anthonappa, R., and King, N. (2018). Eye-tracking in dentistry: what do children notice in the dentist? J. Dent. 78, 72–75. doi: 10.1016/j.jdent.2018.08.006
Chawarska, K., MacAri, S., and Shic, F. (2012). Context modulates attention to social scenes in toddlers with autism. J. Child Psychol. Psychiatry 53, 903–913. doi: 10.1111/j.1469-7610.2012.02538.x
Cheng, K., and Pulo, K. (2003). “Direct interaction with large-scale display systems using infrared laser tracking devices,” in Proceedings of the Asia-Pacific symposium on Information visualisation, Darlinghurst, 67–74.
Clay, V., König, P., and König, S. (2019). Eye tracking in virtual reality. J. Eye Mov. Res. 12:3. doi: 10.16910/jemr.12.1.3
De Lucio, J. V., Mohamadian, M., Ruiz, J. P., Banayas, J., and Bernaldez, F. G. (1996). Visual landscape exploration as revealed by eye movement tracking. Landsc. Urban Plan. 34, 135–142. doi: 10.1016/0169-2046(95)00208-1
Deubel, H., and Schneider, W. X. (1996). Saccade target selection and object recognition: evidence for a common attentional mechanism. Vision Res. 36, 1827–1837. doi: 10.1016/0042-6989(95)00294-4
Duchowski, A. (2003). Eye Tracking Methodology: Theory and Practice. Berlin: Springer. doi: 10.1007/978-1-4471-3750-4.
Dupont, L. Pihel, J., Ode, A., and Van Eetvelde, V. W. (2013). Analyzing the Perception of Water Surfaces in Urban Landscapes using Eye Tracking. Available online at: http://lib.ugent.be/catalog/pug01:4301148 (accessed April 1, 2021).
Eckstein, M. K., Guerra-Carrillo, B., Miller Singley, A. T., and Bunge, S. A. (2017). Beyond eye gaze: what else can eyetracking reveal about cognition and cognitive development? Dev. Cogn. Neurosci. 25, 69–91. doi: 10.1016/j.dcn.2016.11.001
Ehinger, K. A., Hidalgo-Sotelo, B., Torralba, A., and Oliva, A. (2009). Modelling search for people in 900 scenes: a combined source model of eye guidance. Vis. Cogn. 17, 945–978. doi: 10.1080/13506280902834720
Faria, A. L., Andrade, A., Soares, L., and Badia, S. B. I. (2016). Benefits of virtual reality based cognitive rehabilitation through simulated activities of daily living: a randomized controlled trial with stroke patients. J. Neuroeng. Rehabil. 13:96. doi: 10.1186/s12984-016-0204-z
Fove Inc (2017). Fove Inc. Available online at: https://www.getfove.com/ [accessed September 14, 2021].
Frank, J. R., and Danoff, D. (2007). The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med. Teach. 29, 642–647. doi: 10.1080/01421590701746983
Fridman, I., Ubel, P. A., and Higgins, E. T. (2018). Eye-tracking evidence shows that non-fit messaging impacts attention, attitudes and choice. PLoS One 13:e0205993. doi: 10.1371/journal.pone.0205993
Funke, G., Greenlee, E., Carter, M., Dukes, A., Brown, R., and Menke, L. (2016). “Which eye tracker is right for your research? Performance evaluation of several cost variant eye trackers,” in Proceedings of the Human Factors and Ergonomics Society, Washington, DC, 1239–1243. doi: 10.1177/1541931213601289
Gergle, D., Kraut, R. E., and Fussell, S. R. (2013). Using visual information for grounding and awareness in collaborative tasks. Hum. Comput. Interact. 28, 1–39. doi: 10.1080/07370024.2012.678246
Goldberg, P., Sümer, Ö., Stürmer, K., Wagner, W., Göllner, R., Gerjets, P., et al. (2021). Attentive or not? Toward a machine learning approach to assessing students’ visible engagement in classroom instruction. Educ. Psychol. Rev. 33, 27–49.
Gorbunovs, A. (2021). The review on eye tracking technology application in digital learning environments. Balt. J. Mod. Comput. 9, 1–24. doi: 10.22364/bjmc.2021.9.1.01
Haataja, E., Salonen, V., Laine, A., Toivanen, M., and Hannula, M. S. (2021). The relation between teacher-student eye contact and teachers’ interpersonal behavior during group work: a multiple-person gaze-tracking case study in secondary mathematics education. Educ. Psychol. Rev. 33, 51–67. doi: 10.1007/s10648-020-09538-w
Hansen, D. W., and Ji, Q. (2010). In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans. Pattern Anal. Mach. Intell. 32, 478–500. doi: 10.1109/TPAMI.2009.30
Helbing, J., Draschkow, D., and Võ, M. L.-H. (2020). Search superiority: goal-directed attentional allocation creates more reliable incidental identity and location memory than explicit encoding in naturalistic virtual environments. Cognition 196:104147. doi: 10.1016/j.cognition.2019.104147
Hess, E. H., and Polt, J. M. (1960). Pupil size as related to interest value of visual stimuli. Science 132,, 349–350. doi: 10.1126/science.132.3423.349
High Tech Computer Corporation HTC Vive Pro Eye (2019). High Tech Computer Corporation. HTC Vive Pro Eye. Available online at: https://www.vive.com/de/product/vivepro-eye/ [accessed September 14, 2021].
Hollander, J. B., Purdy, A., Wiley, A., Foster, V., Jacob, R. J. K., Taylor, H. A., et al. (2019). Seeing the city: using eye-tracking technology to explore cognitive responses to the built environment. J. Urban. 12, 156–171. doi: 10.1080/17549175.2018.1531908
Holmboe, E. S., Call, S., and Ficalora, R. D. (2016). Milestones and competency-based medical education in internal medicine. JAMA Intern. Med. 176, 1601–1602. doi: 10.1001/jamainternmed.2016.5556
Holmqvist, K., and Andersson, R. (2017). Eye-Tracking: A Comprehensive Guide to Methods, Paradigms and Measures. Oxford: Oxford University Press.
Holmqvist, K., Nyström, M., and Mulvey, F. (2012). “Eye tracker data quality: what it is and how to measure it,” in Proceedings of the Eye Tracking Research and Applications Symposium (ETRA), New York, NY, 45–52. doi: 10.1145/2168556.2168563
Huey, E. B. (1968). The Psychology and Pedagogy of Reading.ı Newark, NJ: International Reading Association.
Hußmann, H., and Oechsner, C. (2017). Gaze-Based Interaction using Pursuits in VR Environments. Available online at: https://www.medien.ifi.lmu.de/padabama/publications/pub/khamis_GazeVR/thesis.pdf (accessed April 1, 2021).
Jarodzka, H., Balslev, T., Holmqvist, K., Nyström, M., Scheiter, K., Gerjets, P., et al. (2012). Conveying clinical reasoning based on visual observation via eye-movement modelling examples. Instr. Sci. 40, 813–827. doi: 10.1007/s11251-012-9218-5
Jarodzka, H., Holmqvist, K., and Gruber, H. (2017). Eye tracking in educational science: theoretical frameworks and research agendas. J. Eye Mov. Res. 10, 1–18. doi: 10.16910/jemr.10.1.3
Jarodzka, H., Scheiter, K., Gerjets, P., and van Gog, T. (2010). In the eyes of the beholder: how experts and novices interpret dynamic stimuli. Learn. Instr. 20, 146–154. doi: 10.1016/j.learninstruc.2009.02.019
Jarodzka, H., Skuballa, I., and Gruber, H. (2021). Eye-tracking in educational practice: investigating visual perception underlying teaching and learning in the classroom. Educ. Psychol. Rev. 33, 1–10. doi: 10.1007/s10648-020-09565-7
Jarodzka, H., Van Gog, T., Dorr, M., Scheiter, K., and Gerjets, P. (2013). Learning to see: guiding students’ attention via a Model’s eye movements fosters learning. Learn. Instr. 25, 62–70. doi: 10.1016/j.learninstruc.2012.11.004
Kaakinen, J. K. (2021). What can eye movements tell us about visual perception processes in classroom contexts? Commentary on a special issue. Educ. Psychol. Rev. 33, 169–179. doi: 10.1007/s10648-020-09573-7
Kang, Z., Jeon, J., and Salehi, S. (2020). Eye tracking data analytics in virtual reality training: application in deepwater horizon oil drilling operation. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 64, 821–825. doi: 10.1177/1071181320641191
Kaushanskaya, M., and Marian, V. (2007). Bilingual language processing and interference in bilinguals: evidence from eye tracking and picture naming. Lang. Learn. 57, 119–163. doi: 10.1111/j.1467-9922.2007.00401.x
Khokhar, A., Yoshimura, A., and Borst, C. W. (2019). “Pedagogical agent responsive to eye tracking in educational VR,” in Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (Los Alamitos, CA: IEEE Computer Society), 1018–1019. doi: 10.1109/VR.2019.8797896
Kiefer, P., Giannopoulos, I., Raubal, M., and Duchowski, A. (2017). Eye tracking for spatial research: cognition, computation, challenges. Spat. Cogn. Comput. 17, 1–19. doi: 10.1080/13875868.2016.1254634
Klein, T. M., Drobnik, T., and Grêt-Regamey, A. (2016). Shedding light on the usability of ecosystem services-based decision support systems: an eye-tracking study linked to the cognitive probing approach. Ecosyst. Serv. 19, 65–86. doi: 10.1016/j.ecoser.2016.04.002
Klin, A., Jones, W., Schultz, R., Volkmar, F., and Cohen, D. (2002). Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Arch. Gen. Psychiatry 59, 809–816. doi: 10.1001/archpsyc.59.9.809
Krupinski, E. A. (2005). Visual search of mammographic images: influence of lesion subtlety. Acad. Radiol. 12, 965–969. doi: 10.1016/j.acra.2005.03.071
Krupinski, E. A., Graham, A. R., and Weinstein, R. S. (2013). Characterizing the development of visual search expertise in pathology residents viewing whole slide images. Hum. Pathol. 44, 357–364. doi: 10.1016/j.humpath.2012.05.024
Krupinski, E. A., Tillack, A. A., Richter, L., Henderson, J. T., Bhattacharyya, A. K., Scott, K. M., et al. (2006). Eye-movement study and human performance using telepathology virtual slides. Implications for medical education and differences with experience. Hum. Pathol. 37, 1543–1556. doi: 10.1016/j.humpath.2006.08.024
Kudelka, V., and Dobesova, Z. (2015). “Eye-tracking testing of GIS interfaces,” in Proceedings of the International Multidisciplinary Scientific Geoconference Surveying Geology and Mining Ecology Management, SGEM, Albena, 585–592.
Lai, M.-L., Tsai, M.-J., Yang, F.-Y., Hsu, C.-Y., Liu, T.-C., Lee, S. W. Y., et al. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educ. Res. Rev. 10, 90–115. doi: 10.1016/j.edurev.2013.10.001
Litchfield, D., Ball, L. J., Donovan, T., Manning, D. J., and Crawford, T. (2010). Viewing another person’s eye movements improves identification of pulmonary nodules in chest x-ray inspection. J. Exp. Psychol. Appl. 16, 251–262. doi: 10.1037/a0020082
Maggio, M. G., De Luca, R., Molonia, F., Porcari, B., Destro, M., Casella, C., et al. (2019a). Cognitive rehabilitation in patients with traumatic brain injury: a narrative review on the emerging use of virtual reality. J. Clin. Neurosci. 61, 1–4. doi: 10.1016/j.jocn.2018.12.020
Maggio, M. G., Latella, D., Maresca, G., Sciarrone, F., Manuli, A., Naro, A., et al. (2019b). Virtual reality and cognitive rehabilitation in people with stroke: an overview. J. Neurosci. Nurs. 51, 101–105. doi: 10.1097/JNN.0000000000000423
Maggio, M. G., Maresca, G., De Luca, R., Stagnitti, M. C., Porcari, B., Ferrera, M. C., et al. (2019c). The growing use of virtual reality in cognitive rehabilitation: fact, fake or vision? A scoping review. J. Natl. Med. Assoc. 111, 457–463. doi: 10.1016/j.jnma.2019.01.003
Maggio, M. G., Russo, M., Cuzzola, M. F., Destro, M., La Rosa, G., Molonia, F., et al. (2019d). Virtual reality in multiple sclerosis rehabilitation: a review on cognitive and motor outcomes. J. Clin. Neurosci. 65, 106–111. doi: 10.1016/j.jocn.2019.03.017
Maksimenko, N., Okolzina, A., Vlasova, A., Tracey, C., and Kurushkin, M. (2021). Introducing atomic structure to first-year undergraduate chemistry students with an immersive virtual reality experience. J. Chem. Educ. 98, 2104–2108. doi: 10.1021/acs.jchemed.0c01441
Mason, L., Pluchino, P., and Tornatora, M. C. (2015). Eye-movement modeling of integrative reading of an illustrated text: effects on processing and learning. Contemp. Educ. Psychol. 41, 172–187. doi: 10.1016/j.cedpsych.2015.01.004
McNamara, A., and Jain, E. (2019). “Eye tracking and virtual reality,” in Proceedings of the SIGGRAPH Asia 2019 Courses SA ’19 (New York, NY: Association for Computing Machinery). doi: 10.1145/3355047.3359424
Merali, N., Veeramootoo, D., and Singh, S. (2019). Eye-tracking technology in surgical training. J. Investig. Surg. 32, 587–593. doi: 10.1080/08941939.2017.1404663
Mercan, E., Aksoy, S., Shapiro, L. G., Weaver, D. L., Brunyé, T. T., and Elmore, J. G. (2016). Localization of diagnostically relevant regions of interest in whole slide images: a comparative study. J. Digit. Imaging 29, 496–506. doi: 10.1007/s10278-016-9873-1
Muna, K., and Bahit, M. (2020). Eye-tracking and metacognitive skills: a review on the use of eye-tracking for measuring students’ metacognitive skills in chemistry learning. J. Phys. Conf. Ser. 1422:012033. doi: 10.1088/1742-6596/1422/1/012033
Nagarkar, D. B., Mercan, E., Weaver, D. L., Brunyé, T. T., Carney, P. A., Rendi, M. H., et al. (2016). Region of interest identification and diagnostic agreement in breast pathology. Mod. Pathol. 29, 1004–1011. doi: 10.1038/modpathol.2016.85
Nakhaeizadeh, S., Morgan, R. M., Olsson, V., Arvidsson, M., and Thompson, T. (2020). The value of eye-tracking technology in the analysis and interpretations of skeletal remains: a pilot study. Sci. Justice 60, 36–42. doi: 10.1016/j.scijus.2019.08.005
Nalanagula, D., Greenstein, J. S., and Gramopadhye, A. K. (2006). Evaluation of the effect of feedforward training displays of search strategy on visual search performance. Int. J. Ind. Ergon. 36, 289–300. doi: 10.1016/j.ergon.2005.11.008
Nasca, T. J., Philibert, I., Brigham, T., and Flynn, T. C. (2012). The next GME accreditation system - rationale and benefits. N. Engl. J. Med. 366, 1051–1056. doi: 10.1056/NEJMsr1200117
Nielsen, J. (2006). F-Shaped Pattern For Reading Web Content (Original Study). Available online at: https://www.nngroup.com/articles/f-shaped-pattern-reading-web-content-discovered/ [accessed October 21, 2021].
Noland, R. B., Weiner, M. D., Gao, D., Cook, M. P., and Nelessen, A. (2017). Eye-tracking technology, visual preference surveys, and urban design: preliminary evidence of an effective methodology. J. Urban. 10, 98–110. doi: 10.1080/17549175.2016.1187197
Norbury, C. F., Brock, J., Cragg, L., Einav, S., Griffiths, H., and Nation, K. (2009). Eye-movement patterns are associated with communicative competence in autistic spectrum disorders. J. Child Psychol. Psychiatry Allied Discip. 50, 834–842. doi: 10.1111/j.1469-7610.2009.02073.x
Orlosky, J., Huynh, B., and Hollerer, T. (2019). “Using eye tracked virtual reality to classify understanding of vocabulary in recall tasks,” in Proceedings of the IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR, San Diego, CA, 66–73. doi: 10.1109/AIVR46125.2019.00019
Paletta, L., Dini, A., and Pszeida, M. (2020). “Emotion measurement from attention analysis on imagery in virtual reality,” in Proceedings of the AHFE 2019 International Conference on Affective and Pleasurable Design, Washington, DC. doi: 10.1007/978-3-030-20441-9_2
Parish-Morris, J., Mahajan, N., Hirsh-Pasek, K., Golinkoff, R. M., and Collins, M. F. (2013). Once upon a time: parent-child dialogue and storybook reading in the electronic era. Mind Brain Educ. 7, 200–211. doi: 10.1111/mbe.12028
Pärnamets, P., Johansson, P., Hall, L., Balkenius, C., Spivey, M. J., and Richardson, D. C. (2015). Biasing moral decisions by exploiting the dynamics of eye gaze. Proc. Natl. Acad. Sci. U.S.A. 112, 4170–4175. doi: 10.1073/pnas.1415250112
Pernice, K., Whitenton, K., and Nielsen, J. (2014). How People Read on the Web: The Eyetracking Evidence. Fremont, CA: Nielsen Norman Group.
Peshkovskaya, A. G., Babkina, T. S., Myagkov, M. G., Kulikov, I., Ekshova, K. V., and Harriff, K. (2017). The socialization effect on decision making in the Prisoner’s Dilemma game: an eye-tracking study. PLoS One 12:e0175492. doi: 10.1371/journal.pone.0175492
Pilgrim, M. J., and Pilgrim, J. (2016). The use of virtual reality tools in the reading-language arts classroom. Texas J. Lit. Educ. 4, 90–97.
Piotrowski, P., and Nowosielski, A. (2020). “Gaze-based interaction for VR environments,” in Proceedings of the Image Processing and Communications. IP&C 2019. Advances in Intelligent Systems and Computing, Vol. 1062, eds M. Choraś and R. Choraś (Cham: Springer). doi: 10.1007/978-3-030-31254-1_6
Poole, A., and Ball, L. J. (2005). “Eye tracking in HCI and usability research,” in Encyclopedia of Human Computer Interaction, ed. C. Ghaoui (Hershey, PA: IGI), 211–219. doi: 10.4018/978-1-59140-562-7.ch034
Potocka, I. (2013). The lakescape in the eyes of a tourist. Quaest. Geogr. 32, 85–97. doi: 10.2478/quageo-2013-0018
Pottle, J. (2019). Virtual reality and the transformation of medical education. Future Healthc. J. 6, 181–185. doi: 10.7861/fhj.2019-0036
Radianti, J., Majchrzak, T. A., Fromm, J., and Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: design elements, lessons learned, and research agenda. Comput. Educ. 147:103778. doi: 10.1016/j.compedu.2019.103778
Rappa, N. A., Ledger, S., Teo, T., Wai Wong, K., Power, B., and Hilliard, B. (2019). The use of eye tracking technology to explore learning and performance within virtual reality and mixed reality settings: a scoping review. Interact. Learn. Environ. doi: 10.1080/10494820.2019.1702560
Reichenberger, J., Pfaller, M., and Mühlberger, A. (2020). Gaze behavior in social fear conditioning: an eye-tracking study in virtual reality. Front. Psychol. 11:35. doi: 10.3389/fpsyg.2020.00035
Rigby, D., Vass, C., and Payne, K. (2020). Opening the ‘Black Box’: an overview of methods to investigate the decision-making process in choice-based surveys. Patient 13, 31–41. doi: 10.1007/s40271-019-00385-8
Rosa, P. J., Gamito, P., Oliveira, J., Morais, D., Pavlovic, M., Smyth, O., et al. (2017). Eye movement analysis and cognitive assessment: the use of comparative visual search tasks in a non-immersive vr application. Methods Inf. Med. 56, 112–116. doi: 10.3414/ME16-02-0006
Rutkowski, S., Rutkowska, A., Racheniuk, H., Szczepańska-Gieracha, J., and Szczegielniak, J. (2019). “Short-term exercise training in virtual reality in patients with COPD solution or white elephant?,” in Proceedings of the ERS International Congress, Madrid.
Sadasivan, S., Greenstein, J. S., Gramopadhye, A. K., and Duchowski, A. T. (2005). “Use of eye movements as feedforward training for a synthetic aircraft inspection task,” in Proceedings of the Technology, Safety, Community: Conference on Human Factors in Computing Systems, Portland, OR, 141–149. doi: 10.1145/1054972.1054993
Saxena, N., Kyaw, B. M., Vseteckova, J., Dev, P., Paul, P., Lim, K. T. K., et al. (2016). Virtual reality environments for health professional education. Cochrane Database Syst. Rev. 2016:CD012090. doi: 10.1002/14651858.CD012090
Schlechtinger, M. (2020). What are you Looking at? Using Eye Tracking to Improve Learning in Virtual Environments. Siegen: University of Siegen, 54–59. doi: 10.25819/ubsi/2762
Seidel, T., Schnitzler, K., Kosel, C., Stürmer, K., and Holzberger, D. (2021). Student characteristics in the eyes of teachers: differences between novice and expert teachers in judgment accuracy, observed behavioral cues, and gaze. Educ. Psychol. Rev. 33, 69–89. doi: 10.1007/s10648-020-09532-2
Shic, F., Bradshaw, J., Klin, A., Scassellati, B., and Chawarska, K. (2011). Limited activity monitoring in toddlers with autism spectrum disorder. Brain Res. 1380, 246–254. doi: 10.1016/j.brainres.2010.11.074
Simpson, J. G., Furnace, J., Crosby, J., Cumming, A. D., Evans, P. A., Ben David, F., et al. (2002). The Scottish doctor - Learning outcomes for the medical undergraduate in Scotland: a foundation for competent and reflective practitioners. Med. Teach. 24, 136–143. doi: 10.1080/01421590220120713
Smith, S. M., and Krajbich, I. (2019). Gaze amplifies value in decision making. Psychol. Sci. 30, 116–128. doi: 10.1177/0956797618810521
Spinks, J., and Mortimer, D. (2016). Lost in the crowd? Using eye-tracking to investigate the effect of complexity on attribute non-attendance in discrete choice experiments. BMC Med. Inform. Decis. Mak. 16:14. doi: 10.1186/s12911-016-0251-1
Stein, R., and Brennan, S. E. (2004). “Another person’s eye gaze as a cue in solving programming problems,” Proceedings of the Sixth International Conference on Multimodal Interfaces, New York, NY, 9–15. doi: 10.1145/1027933.1027936
Strohmaier, A. R., MacKay, K. J., Obersteiner, A., and Reiss, K. M. (2020). Eye-tracking methodology in mathematics education research: a systematic literature review. Educ. Stud. Math. 104, 147–200. doi: 10.1007/s10649-020-09948-1
Sun, Y., Li, Q., Zhang, H., and Zou, J. (2018). The Application of Eye Tracking in Education. New York, NY: Springer International Publishing, 27–33. doi: 10.1007/978-3-319-63859-1_4
Suvorov, R. (2015). The use of eye tracking in research on video-based second language (L2) listening assessment: a comparison of context videos and content videos. Lang. Test. 32, 463–483. doi: 10.1177/0265532214562099
Swing, S. R. (2007). The ACGME outcome project: retrospective and prospective. Med. Teach. 29, 648–654. doi: 10.1080/01421590701392903
Tanriverdi, V., and Jacob, R. J. K. (2000). “Interacting with eye movements in virtual environments,” in Proceedings of the Conference on Human Factors in Computing Systems, New York, NY, 265–272. doi: 10.1145/332040.332443
Tatler, B. W., Wade, N. J., Kwan, H., Findlay, J. M., and Velichkovsky, B. M. (2010). Yarbus, eye movements, and vision. Iperception 1, 7–27. doi: 10.1068/i0382
Tenenbaum, E. J., Amso, D., Abar, B., and Sheinkopf, S. J. (2014). Attention and word learning in autistic, language delayed, and typically developing children. Front. Psychol. 5:490. doi: 10.3389/fpsyg.2014.00490
Tien, T., Pucher, P. H., Sodergren, M. H., Sriskandarajah, K., Yang, G.-Z., and Darzi, A. (2014). Eye tracking for skills assessment and training: a systematic review. J. Surg. Res. 191, 169–178. doi: 10.1016/j.jss.2014.04.032
Torralba, A., Oliva, A., Castelhano, M. S., and Henderson, J. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychol. Rev. 113, 766–786. doi: 10.1037/0033-295X.113.4.766
Ustun, A., Yılmaz, R., and Karaoðlan Yılmaz, F. G. (2020). “Virtual Reality in Medical Education,” in Mobile Devices and Smart Gadgets in Medical Sciences, Hershey, PA: IGI Global, 56–73. doi: 10.4018/978-1-7998-2521-0.ch004
van Gog, T., Jarodzka, H., Scheiter, K., Gerjets, P., and Paas, F. (2009). Attention guidance during example study via the model’s eye movements. Comput. Human Behav. 25, 785–791. doi: 10.1016/j.chb.2009.02.007
Vandenplas, J. R., Herrington, D. G., Shrode, A. D., and Sweeder, R. D. (2021). Use of simulations and screencasts to increase student understanding of energy concepts in bonding. J. Chem. Educ. 98, 730–744. doi: 10.1021/acs.jchemed.0c00470
Velichkovsky, B. M. (1995). Communicating attention: gaze position transfer in cooperative problem solving. Pragmat. Cogn. 3, 199–223. doi: 10.1075/pc.3.2.02vel
Voisin, S., Pinto, F., Morin-Ducote, G., Hudson, K. B., and Tourassi, G. D. (2013). Predicting diagnostic error in radiology via eye-tracking and image analytics: preliminary investigation in mammography. Med. Phys. 40:101906. doi: 10.1118/1.4820536
Weiss, S., Bongartz, H., Boll, S., and Heuten, W. (2018). “Applications of immersive virtual reality in nursing education - a review,” in Proceedings of the 1 Clusterkonferenz Zukunft der Pflege – Innovative Technologien für die PraxisAt, Oldenburg.
Yarbus, A. L. (1967). Eye Movements and Vision. Boston, MA: Springer. doi: 10.1007/978-1-4899-5379-7
Yoshimura, A., Khokhar, A., and Borst, C. W. (2019). “Eye-gaze-triggered visual cues to restore attention in educational VR,” in Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (Washington, DC: IEEE Computer Society), 1255–1256. doi: 10.1109/VR.2019.8798327
Keywords: eye-tracking, virtual reality, education and VR, eye-tracking in VR, education
Citation: Mikhailenko M, Maksimenko N and Kurushkin M (2022) Eye-Tracking in Immersive Virtual Reality for Education: A Review of the Current Progress and Applications. Front. Educ. 7:697032. doi: 10.3389/feduc.2022.697032
Received: 18 April 2021; Accepted: 01 February 2022;
Published: 10 March 2022.
Edited by:
Christothea Herodotou, The Open University, United KingdomReviewed by:
Mark Billinghurst, University of South Australia, AustraliaSergio Rampino, Normal School of Pisa, Italy
Copyright © 2022 Mikhailenko, Maksimenko and Kurushkin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Mikhail Kurushkin, kurushkin@scamt-itmo.ru