Bruno Seraglia1 Luciano Gamberini1* Konstantinos Priftis1,3 Pietro Scatturin2 Massimiliano Martinelli1 Simone Cutini3
- 1Department of General Psychology, University of Padua, Padua, Italy
- 2Department of Developmental Psychology and Socialization, University of Padua, Padua, Italy
- 3Laboratory of Neuropsychology, IRCCS San Camillo Hospital, Lido-Venice, Italy
For over two decades Virtual Reality (VR) has been used as a useful tool in several fields, from medical and psychological treatments, to industrial and military applications. Only in recent years researchers have begun to study the neural correlates that subtend VR experiences. Even if the functional Magnetic Resonance Imaging (fMRI) is the most common and used technique, it suffers several limitations and problems. Here we present a methodology that involves the use of a new and growing brain imaging technique, functional Near-infrared Spectroscopy (fNIRS), while participants experience immersive VR. In order to allow a proper fNIRS probe application, a custom-made VR helmet was created. To test the adapted helmet, a virtual version of the line bisection task was used. Participants could bisect the lines in a virtual peripersonal or extrapersonal space, through the manipulation of a Nintendo Wiimote® controller in order for the participants to move a virtual laser pointer. Although no neural correlates of the dissociation between peripersonal and extrapersonal space were found, a significant hemodynamic activity with respect to the baseline was present in the right parietal and occipital areas. Both advantages and disadvantages of the presented methodology are discussed.
Introduction
Virtual Reality in Cognitive Neuroscience
Nowadays, Virtual Reality (VR) is a useful tool in several fields and has reached a considerable value in medical, psychological, and neuropsychological treatments (Wiederhold and Wiederhold, 2000; Merians et al., 2006; Optale et al., 2010; Tanaka et al., 2010; Tomikawa et al., 2010). Because VR interactions, however, take place within artificial worlds, it is difficult to compare, analyze, and interpret results with those obtained during real-life experiences. Together with the quantitative data registered during the interaction of participants with the synthetic environment, qualitative data obtained through questionnaires and self-report measures represent valuable methods but, in the same time, are not sufficient to support completely this advanced and useful tool (i.e., VR).
Studies employing recent brain imaging techniques have shown that immersive VR interactions activate the same brain areas as those activated in the corresponding real situation (Campbell et al., 2009; Clemente et al., 2010). Nonetheless, there is also evidence that different brain regions are activated when participants observe real objects than when they observe virtual ones (Decety et al., 1994; Perani et al., 2001).
Nowadays, the most commonly used brain imaging techniques are functional Magnetic Resonance Imaging (fMRI; deCharms, 2008), Electroencephalography (EEG; Niedermeyer and Lopes da Silva, 2004), and Positron Emission Tomography (PET; Ter-Pogossian et al., 1975). fMRI monitors brain activity, by using blood oxygen level dependent (BOLD) responses. EEG records electrical activity along the scalp produced by the firing of neurons within the brain. PET detects gamma rays emitted by a tracer, which is injected into the body by means of a biologically active molecule. A major problem in utilizing these brain imaging techniques to study the neural correlates of VR experiences, especially with reference to fMRI and PET, regards the “immersion” within the artificial environment. Huge machinery dimensions, disturbing noise, and electro-magnetic interferences with other instrumentations, together with the horizontal and unnatural position of the participant during scans' acquisition, constitute the most limiting factors in the use of the aforementioned techniques with VR paradigms. The studies that we report here are divided in non-immersive and immersive ones, accordingly to the brain imaging technique used. Non-immersive techniques are those in which the virtual environment is visualized through normal desktop monitors. Immersive techniques are those in which Head Mounted Displays (HMDs), 3D glasses, or similar equipments are used to visualize the virtual environment.
VR applications and studies with fMRI vary in different disciplines (for a review, see Wiederhold and Wiederhold, 2008). Astur et al. (2004), by administering a virtual version of the eight-arm radial maze task to normal adults, found a bilateral activation of the hippocampus, which is responsible of three-dimensional spatial memory (Iaria et al., 2003; Parslow et al., 2005). Calhoun et al. (2004) observed the activation of several separate, brain networks when participants had to drive a simulated car under the effect of alcohol. With respect to participants who were affected by alcohol, participants who were not affected by alcohol showed specific activations in the orbitofrontal (OF) and motor regions, whereas visual and medial frontal regions were not activated. Overall, fMRI scans showed that alcohol intoxication may affect OF/anterior cingulate areas as well as motor and cerebellar regions (Calhoun et al., 2005; Jeong et al., 2006; Meda et al., 2009). Slobounov et al. (2006) were the first to identify which brain areas were activated in response to a VR visual field motion, when participants experienced ego-motion (i.e., the actual motion of the body as a response to the presence of optic flow) and vection (i.e., the illusory self-motion following a moving background). Largest activations were found within the area V5 bilaterally and the superior temporal sulcus, which are known to be specifically involved in the perception of biological motion.
In the abovementioned studies, participants visualized virtual scenarios through mirror reflection of an LCD screen placed outside the fMRI machine. Thus, participants did not have a full immersion into the virtual environment. Hoffman et al. (2004) have developed a system which uses prototype VR goggles inside the fMRI machine. This study explored the neural correlates of VR during pain situations. Hoffman et al. found that pain-related brain activity was significantly reduced in the VR condition with respect to the non-VR one, in all regions of interest: the anterior cingulate cortex, the primary (S1) and secondary (S2) somatosensory cortices, the insula, and the thalamus. Lee et al. (2005) investigated which brain areas of smokers are activated when smoking-related cues are compared with neutral ones. fMRI scans showed an increased activation in the smoking-related cues condition, principally in the prefrontal cortex (PFC) and in the left anterior cingulate cortex (ACC), consistently with the findings of previous studies (Brody et al., 2004). Note that stereoscopic MR-compatible goggles were also used in this study.
A field of interest is that regarding the realization of specific devices and tools in order to avoid electromagnetic interferences during fMRI registration caused by surrounding or supplementary equipment. Important factors that influence MRI registration are: the distance between the imaging region and other electronic components (e.g., cables, sensors, or transducers), the shielding system, the filtering system, etc., (Gassert et al., 2008). The development, validation, and testing of a prototype force-feedback device for VR-fMRI was the topic of the study by Di Diodato et al. (2007). In the testing experiment the participants had to touch a virtual object in two different conditions: one with force-feedback and one with no-force-feedback. In the force-feedback condition, higher brain activation was recorded in the left S1, in the supplementary somatosenory cortex bilaterally (S2), and in the posterior insula. An adaptation of the Phantom Premium 1.5 haptic device has been created to perform grasping operations in virtual environments during fMRI recordings (Hribar et al., 2009). Moreover, Resonance Technology Inc. (http://www.mrivideo.com) has recently developed VR glasses compatible with the fMRI machinery.
EEG remains a useful tool for studying the neurophysiological correlates of VR experiences (Pugnetti et al., 1996). For instance, EEG has been used to study the neurophysiological correlates during a non-interactive VR experience where a group of adolescents was requested to watch roller coaster rides. The aim of that study was to measure spatial presence during the visual processing. The results showed an activation of the parietal areas responsible for spatial navigation (Baumgartner et al., 2006). The aim of a recent study was to explore whether specific brain activation patterns were associated with driving at excessive speed (Jancke et al., 2008). In this study a virtual driving simulator was used to simulate realistic driving conditions. Compared with normal driving in which traffic rules were strictly followed, during excessively fast driving, activation of the right lateral PFC was increased in the α-band activity. Bischof and Boulanger (2003) measured EEG activity while healthy normal participants navigated through virtual mazes using desktop screens. They found a positive relation between the frequency of theta episodes and the difficulty of maze navigation. After immersive VR therapy (VRT) was conducted, a group of 20 alcohol-dependent participants reduced the craving for alcohol, by increasing alpha wave activity in the frontal areas (Lee et al., 2009).
In one of the first studies that used PET to investigate neural correlates during a virtual navigation task, the authors found that navigation in humans is supported by a network of brain areas: the right hippocampus, the right caudate nucleus, the right inferior parietal lobule, and the medial parietal regions (Maguire et al., 1998; Jeong et al., 2006). Horikawa et al. (2005) used PET to identify brain areas involved in a simulated driving task. They found activation in the thalamus, the midbrain, the cerebellum, and the posterior cingulate gyrus, suggesting an involvement of these areas in the maintenance of driving performance.
In the next paragraph, we present a new and growing methodology to study real-time brain areas activation called functional Near-infrared Spectroscopy (fNIRS), while it was used during a VR experiment. Together with low-cost, portability, and reasonable spatial resolution, one of the positive characteristic of fNIRS is that it permits the use of an adapted HMD to grant a full immersion into the virtual environment.
fNIRS and Cognitive Neuroscience
fNIRS is a promising brain imaging technique that allows researchers to localize and measure cerebral blood flow and oxygenation. It is a real-time, diagnostic, and non-invasive technique, capable of measuring tissue oxygenation StO2, using low-cost and portable instrumentation. The fNIRS uses optical radiation (i.e., photons which wavelength is near the infrared range [NIR]: 700–950 nm). The fNIRS probes have light sources, which penetrate tissues, and detectors (optical fibers), which detect light radiations leaked from the biological tissue after having completed a depth and bended variable path (characterized by multiple scattering events, see below) on the same side of the light source. The typical inter-optode distance ranges between 2.5 and 4 cm; this permits near-infrared light to penetrate biological tissue of about 1.5–2.5 cm in depth. Near-infrared light in the biological tissue undergoes two main processes which are both wavelength dependent: scattering and absorption. Scattering, a process where the light is forced to deviate from a rectilinear trajectory because of the resistance of the tissue, is measured by the scattering coefficient (μs). On the other hand, biological tissue absorption, a process that is due to the presence of hemoglobin and because of which the light is retained by the tissue, is measured by the absorption coefficient (μa). Oxygenate hemoglobin (HbO) and deoxygenate hemoglobin (HbR) have different NIR absorption rates. This difference allows the separate measurement of the two Hb types and, thus, the oxygen saturation (StO2; i.e., the quantification of the ratio of HbO with respect to the total amount of hemoglobin in the microcirculation). After the neural activation of a cluster of neurons in a given region, the HbO concentration in this region increases to supply the additional oxygen demand of active neurons, and HbR concentration decreases almost simultaneously (the phenomenon that relates neural activity with hemodynamic activity is called neurovascular coupling, see Villringer and Dirnagl, 1995; note that fMRI inference is also based on neurovascular coupling). The typical HbO response function is characterized by a sluggish temporal profile when compared to that of neural activity: usually, hemodynamic activity begins to increase after about 1 s following changes in neural activity, it reaches its peak in around 5–7 s after neural activity, and it slowly returns to baseline activity after 12–15 s. Most fNIRS instrumentations allow the emission of intensity constant and continuous light, capable of obtaining relative measures of StO2. The fNIRS technologies are in continuous evolution. The improvement of spatial resolution, for example, remains one of the most studied problems and several solutions have been suggested (Owen-Reece et al., 1999). Furthermore, the statistical interpretation of fNIRS data has been developed and refined over the years (Villringer and Chance, 1997; Strangman et al., 2002), together with the constant improvement of methods for eliminating artifacts automatically (Hoshi, 2005).
The fNIRS applications in neurosciences have explored the neural correlates of several human behaviors (sensorimotor apparatus: Miyai et al., 2001; Shimada et al., 2004; visual perception: Taga et al., 2003; Schroeter et al., 2004; executive functions: Schroeter et al., 2002; Hoshi et al., 2003).
There are several advantages of fNIRS, compared to other imaging techniques (e.g., fMRI, PET). First, fNIRS is easily portable and has lower cost. Second, fNIRS allows the participants to be almost totally free in their movements. Third, because it uses only light sources, fNIRS is completely non-invasive. Fourth, fNIRS is characterized by a temporal resolution of 100 Hz or higher, that is significantly greater than those of fMRI and PET. Finally, by being interference-free, fNIRS permits simultaneous multiple registrations with fMRI, PET, and EEG. Indeed, fNIRS represents one of the most useful brain imaging techniques for replicating and validating data and registrations acquired with fMRI (Hoshi, 2005; Huppert et al., 2006; Irani et al., 2007).
The fNIRS, however, has some limitations. First, only cortical activity can be examined with fNIRS, given the limited depth penetration of near-infrared light into the skull. Second, the anatomical information cannot be directly inferred by fNIRS, thus it must be obtained with the help of other techniques. Third, hair can interfere with the transmission of light among the source, the scalp, and the detector. Finally, given that the participants are almost totally free to move, the holder must be fixed on the head in order to ensure a correct registration and to avoid motion artifacts (Hoshi, 2005; Huppert et al., 2006; Irani et al., 2007).
To the best of our knowledge to date there are no attempts to study the neural correlates of immersive VR experience with fNIRS. Previous studies have implemented VR but only in a non-immersive desktop setting. Combe et al. (2010), for example, have studied the neural correlates of depth perception estimation in a 3D environment, and its effects on the participants' emotional state. Previous studies have implemented flight-simulators (Takeuchi, 2000), drive-simulators (Li et al., 2009), or war video, game-like simulations (Izzetoglu et al., 2003).
In our study, we elaborated more on the methodology used in a previous study (Gamberini et al., 2008), in order to find a valid solution concerning correct signal acquisition with fNIRS during an immersive VR task. Using a line bisection task, we expected that both the parietal lobe and the parieto-occipital junction would be activated (Weiss et al., 2000). Indeed, by using PET, Weiss et al. (2000) asked participants to perform a line bisection task at two distances, 70 cm (near space) and 170 cm (far space). Lines were presented on the center of a monitor. Participants performed bisection using a laser pointer. Weiss et al. found that when lines were bisected in near space, the dorsal visuomotor stream was activated (i.e., the dorsal occipital cortex and the parietal cortex along the intraparietal sulcus). In contrast, when lines were bisected in far space, the ventral visuoperceptual stream was activated (i.e., the ventral occipital cortex bilaterally and the right medial temporal cortex).
In the present study we investigated the possibility to implement fNIRS while participants experienced immersive VR. Our solution involved the use of a modified VR helmet to allow fNIRS optical fibers to be placed correctly over the scalp. Therefore, the aim of the study was to investigate whether this technical apparatus is suitable for correct signal acquisition with fNIRS. We employed the line bisection task which represents an efficient and valid paradigm to test brain areas activation during spatial attention deployment. Indeed, the line bisection task is a widely used one. Although this is a relatively simple task, it gives remarkable results in terms of visuo-spatial attention and perception. It is widely accepted and several studies have reported similar results, in many of its variations (Jewell and McCourt, 2000 for a review; Ferber and Karnath, 2001; Longo and Lourenco, 2006; Rorden et al., 2006). Very few studies have simulated the line bisection task in an immersive virtual environment (Baheux et al., 2006; Garrison and Ellard, 2009). Therefore, our aim was to employ this paradigm to test the feasibility of our technical proposal for registering fNIRS signals in an immersive VR environment.
Method
Participants
Eight right-handed, students at the University of Padua (7 males; mean age: 27.6 years, range: 24–36) participated in the experiment after providing their informed consent. All participants had normal or corrected-to-normal vision, and normal color vision. None of the participants reported a prior history of neurological or psychiatric disorders, and none was under medication at the time of testing. One participant was excluded from the data analysis because his fNIRS signal was too noisy.
Apparatus and Stimuli
Participants saw the virtual environment through an adaptation of a V8 Research HMD (Dual 1.3″ diagonal Active Matrix Liquid Crystal Displays; 800 × 600 resolution; 60° diagonal Field of View; 200:1 Contrast ratio). V8 Research HMD LCDs were taken from the original helmet and attached to a modified bike helmet that was modified in order to reach brain areas from the fNIRS optical fibers (see Figure 1). A Velcro belt was attached at the back of the helmet in order to counterbalance the effect of the LCDs' weight in front of the helmet. This belt was subsequently secured to the participants back through a thoracic belt. An Intersense tracker was mounted above the LCDs, in order to allow the participants to completely immerse into the environment (see Figure 1).
Figure 1. The adapted virtual reality helmet. The helmet was created by attaching the LCDs removed from a V8 Research head mounted display to a modified bike helmet. The fNIRS optical fibers were applied to the parietal and occipital areas. A Velcro belt was attached at the back of the helmet in order to counterbalance the effect of the LCDs' weight in front of the helmet. The belt was subsequently secured to the participants back through a thoracic belt.
The virtual environment was created using 3DStudio Max 8.0 for the development of three-dimensional objects and Virtools 3.5 for the interaction with them. A virtual room was created with a “wooden” table in the center. Above and aligned with the table's center, there was a white panel (50 × 50 cm) for displaying horizontal lines. In the panel's center there was a bracket; on the left of the panel there was a telephone, while on the right there were some books (see Figure 2).
Figure 2. The virtual environment. On the left, a panoramic view of the virtual room is presented; the mobile chair with wheels was created to simulate and motivate the movement along the distances of line presentation, 60 and 120 cm. On the right, the participants' point of view of the virtual environment is presented; lines were presented over the white panel and could be bisected moving a red dot through the manipulation of a Nintendo Wiimote® controller.
There were two viewer-line distances: 60 cm and 120 cm. Lines were 4–8 cm-long at the distance of 60 cm, and 8–16 cm-long at the distance of 120 cm (subtending a visual angle of 3.82° and 7.54° for each line pair 4–8 and 8–16, respectively). In front of the table there was a mobile chair with wheels that served for the participant, as he was instructed that, once he was seated, he could move along the two distances through that chair. Virtual lines were planes (i.e., a type of 3D object primitive used in computer graphics) that were 2 mm thick. To guarantee high precision in the response acquisition each centimeter of each line was subdivided in four segments, (0.25 mm each). To decrease normal aliasing provoked by three-dimensional lines, line textures of the same dimensions were superimposed above the virtual lines. To simulate the laser pointer, a Nintendo Wiimote® controller was used. The Wiimote operates as a normal mouse throughout the software GlovePie (http://glovepie.org/). In the virtual environment the Wiimote moved a 2.5 mm red dot as a simulation of the same one represented by a real laser pointer. The virtual red dot was represented by a 3D cone whose tip could collide with the lines, giving the point of contact over them (i.e., the bisection point). The A button on the Wiimote served to memorize the response (the last point of contact over the line) and to switch to another line and/or distance. The virtual environment was perceived stereoscopically as two points of view. That is, two cameras outdistanced by 2 cm were created as a simulation of the left and right eyes: the left camera for the left LCD and the right camera for the right LCD. The task was divided in two blocks: the experimental block and the control block. The experimental block comprised 72 stimuli in which the participant had to bisect the lines. The control block comprised 36 stimuli in which only the right extremity of the lines had to be reached. The intertrial interval between each line presentation (onset) was 12 s in order to register a correct hemodynamic response. Lines and distances presentation was randomized. The order of blocks was counterbalanced across participants.
Procedure
After having compiled the informed consent, each participant read the instructions in order to complete the experimental task. Then the participant was invited to seat in a comfortable chair placed inside a sound-attenuated and dimly-lit room, where the VR helmet and the fNIRS optical fibers were placed on his/her head. The average time to apply the helmet was approximately 20′. Before starting the experiment the participant was instructed to remain steady as much as possible, during the experiment, and to avoid repetitive movements; the experimenter highlighted this instruction to be sure that the participant was aware of the fact that each single movement could interfere with signal registration. The participants performed a training task in which 16 lines were presented (eight for the experimental block and eight for the control block). The overall duration of the experiment was 20′.
fNIRS Data Acquisition
The recording optical unit was a multi-channel frequency-domain NIR spectrometer (ISS Imagent™, Champaign, IL), equipped with 32 laser diodes (16 emitting light at 690 nm, and 16 at 830 nm) modulated at 110 MHz. The diode-emitted light was conveyed to the participant's head by multimode core glass optical fibers (heretofore, sources; OFS Furukawa LOWOH series fibers, 0.37 of numerical aperture) with a length of 250 cm and a core diameter of 400 μ m. Light that scattered through the brain tissue was carried by detector optical fiber bundles (diameter 3 mm) to four photo-multiplier tubes (PMTs; R928 Hamamatsu Photonics). The PMTs were modulated at 110.005 MHz, generating a 5 KHz heterodyning (cross-correlation) frequency. To separate the light as a function of source location, the sources time-shared the four parallel PMTs via an electronic multiplexing device. Only two sources (one per hemisphere) were synchronously (t = 4 ms) active (i.e., emitting light), such that the resulting sampling frequency was f = 15.0625 Hz, due to the 64 ms sampling period required to cycle through the 16 multiplexed channels. To stabilize the optical signal, a dual-period averaging was performed, resulting in a final sampling period of 128 ms (f = 103/128 = 7.8125 Hz). Following detection and consequent amplification by the PMTs, the optical signal was converted into alternating current (AC), direct current (DC), and phase (Φ) signal for each source-detector channel, considering separately each light wavelength. These values were then converted into estimates of μ a variations (Δμα) using the differential-pathlength factor (DPF) method.
Temporal variations (Δ) in the cerebral oxy-hemoglobin (ΔHbO) and deoxy-hemoglobin (ΔHbR) concentrations were calculated based on the values of Δμα at the two wavelengths (Sevick et al., 1991; Franceschini et al., 2000).
The spatial arrangement of source/detector pairs on the scalp was determined using a recent probe placement method (Cutini et al., 2011b). Sources and detectors were held in place on the scalp using a custom holder with Velcro straps. Each source was composed of two source optical fibers (one for each wavelength). The distance between each source/detector pair (i.e., channel) was 30 mm. This probe arrangement included 20 channels, providing 20 measurements for HbO and 20 for HbR. The holder covered partially both the occipital and the parietal lobes, as in a previous fNIRS study adopting the same probe placement criteria (Cutini et al., 2011a). An illustration of the regions covered in the present study is provided in Figure 3 (for details, see Cutini et al., 2011a, b).
Figure 3. The fNIRS recording. In the bottom part of the figure, the location of the regions investigated in the present study, with cerebral projections of detectors (black) and channels (white) superimposed on a template brain (occipital view). The letters of the detectors indicate the lobe (P: parietal; O: occipital) and the hemisphere (L: left; R: right). The number indicates the source. Channels are named according to the source-detector pair: for instance, detector OL and source 5 created the channel OL5. Further details can be found in Cutini et al., (2011a, b). In the top part of figure, hemodynamic response profile in symmetrical channels PR1 and PL1 (i.e., right vs. left parietal lobe) during virtual line bisection. A visual inspection of the response profiles in the two channels suggests a marked difference for what concerns the presence of task-related hemodynamic activity in the two parietal lobes.
fNIRS Data Analysis
Individual hemodynamic responses were baseline-corrected on a trial-by-trial basis by subtracting the mean intensity of the optical signal recorded in the interval 2 s—0 from the onset (i.e., the presentation of the to-be-bisected line) from the overall hemodynamic activity (12 s) (Schroeter et al., 2002). Trials contaminated by artifacts were eliminated using the outlier removal algorithm proposed by Devaraj (2005). The mean value and the difference between the maximum and minimum values (range) were calculated considering all trials in a given condition. The mean value and range were also calculated for each single trial. Single trial mean and range values were then compared with the mean values of all trials in that condition. Trials with a range or mean value greater than the condition mean ±3 standard deviations were discarded from analysis. Signal averaging of all remaining trials in each condition was then performed. Noisy channels (with standard deviation > 700 nM) were discarded from further analysis (less than 5%). The averaged hemodynamic signal was smoothed with a Savitzky and Golay's (1964) filter with polynomial order equal to three and frame size equal to 39 time-points (i.e., 5 s).
Subsequently, the mean ΔHbO and ΔHbR signal intensities during the vascular response (i.e., the peak value reached during the temporal window between 5 and 9 s from onset) were calculated for each participant and condition. The analysis performed on the data recorded in the trials using the individual optical maps aimed at verifying the channels showing a significant activation increase relative to the baseline.
Results
Although a specific functional dissociation between extrapersonal and peripersonal space was not observed, we were able to verify the reliability of the measurements in terms of a significant hemodynamic activity with respect to the baseline. To this scope, we performed a series of t-tests on the individual ΔHbO concentrations observed during the hemodynamic response vs. baseline (pooled activity of peak amplitudes during extrapersonal and peripersonal bisection conditions). We observed statistically significant HbO activity in most of the occipital and parietal channels whereas no significant HbR decrements were observed, mainly because of the poorer signal-to-noise ratio with respect to that of HbO. In particular, we observed a pronounced ΔHbO activation in the right parietal channels (see Figure 3) during virtual line bisection (e.g., right parietal channel 1 (PR1): t(6) = 2.26, p = 0.032; right parietal channel 2 (PR2): t(6) = 2.12, p = 0.037; one-tailed t-tests (see Figure 3 for the location of those channels).
The activity observed in PR1 seemed to be a reliable task-related hemodynamic response; nevertheless, such activity might have been contaminated by physiological components unrelated to the task (i.e., skin blood flow, Takahashi et al., 2011), although the event-related design should have limited this influence. In order to provide a further proof in regard to the reliability of PR1 hemodynamic response, we compared the activity of PR1 with that of the symmetrical channel on the left hemisphere (i.e., PL1). Interestingly, we observed a difference between PR1 and PL1 activity that was very close to significance (p = 0.077, t(6) = 1.628, one tailed t-test). The real version of the line bisection task was found to activate the right parietal areas in several previous studies (Fink et al., 2000, 2001, 2002; Foxe et al., 2003; Hurwitz et al., 2011). Given the exploratory nature of the present investigation, data analysis shown here was meant to represent only a broad verification of the feasibility of our methodology. Nevertheless, further analysis as well as tests on the methodology used should be performed to completely assess the potential and the limitations of the proposed methodology.
Behavioral results of the virtual line bisection task showed a significant difference in the bisection error in the comparison between the performance in peripersonal and extrapersonal space. Although an overall error to the left of the true midpoint was present (M = − 2.49), participants' performance showed a greater error to the left in virtual peripersonal space than in virtual extrapersonal space, t(6) = − 2.112, t(6) = 0.039, one-tailed t-tests (see Figure 4). The discrepancy between the behavioral results, in which a dissociation between peripersonal and extrapersonal space is revealed, and the fNIRS results, in which a dissociation between peripersonal and extrapersonal space is not revealed, could be ascribed to the different nature of the present study and that of Weiss et al. (2000). As stated above, further testing and validation of the system need to be done to completely assess its potential and limitations.
Figure 4. Graph of the behavioral results. The graph shows the percentage error (Y axis) along the distances of line presentation, 60 and 120 cm (X axis). Negative values indicate an error on the left of the true centre of the line; positive values indicate an error on the right of the true centre of the line. Error bars represent the standard error ±SE.
Discussion
Results showed an activation of the parietal and occipital lobes both when the lines were bisected at 60 cm and at 120 cm. These areas are implicated in visuomotor tasks, that is the case of this study in which participants had to perform hand movements (i.e., using the simulated laser pointer), while they were visually focusing attention to find the center of the lines. Several researchers have found parietal activation during attention, space perception and visuo-spatial tasks, and frequently the paradigm used was the line bisection task (Fink et al., 2000, 2001, 2002; Bjoertomt et al., 2001; Foxe et al., 2003; Thiel et al., 2004; Peers et al., 2005; Waberski et al., 2008; Azañón et al., 2010; Hurwitz et al., 2011; Pisella et al., 2011); and in particular a right parietal lobe predominance (Fink et al., 2000, 2001, 2002; Foxe et al., 2003; Hurwitz et al., 2011). On the other hand, there are few evidences when visuo-spatial tasks are performed during VR simulation (Maguire et al., 1998; Baumgartner et al., 2006; Jeong et al., 2006; Hribar et al., 2009). Moreover, no fNIRS and VR studies that find similar results are reported yet. The inability to find dissociation in neural activation for the two distances presented, as for the study of Weiss et al. (2000), could be ascribed to the virtual component of the task. In the present study we used V8 Research LCDs attached on a bike helmet in order to have as much as possible the same conditions of the study by Gamberini et al. (2008). The aim of our exploratory investigation was to study the possibility to implement the fNIRS brain imaging technique in the study of immersive VR experience. Using a virtual line bisection task a cerebral activation was present in the parietal and occipital lobes as expected. Definitely, further investigation is needed to confirm the validity and reliability of our results. Although the experiment was not flawless, it represents a good starting point in the implementation of the fNIRS for the investigation of VR interactions: in this regard, our encouraging results might trigger further research in the field.
A major problem was represented by the application and assembly of the adapted helmet on each participant. As the helmet was not adjustable and each participant has a different head circumference, we had to use Velcro to fix the fNIRS patch both to head and to the helmet, and to add several pieces of rubber to fit as much as possible correctly and comfortably the helmet to the participant. Moreover, in this solution only BOLD in parietal and occipital brain areas can be measured, because for the other ones there was no space to place fNIRS probe. Solving those problems will be the goal of our future studies.
The implementation of less cumbersome HMDs (i-Trek 3D PC, Virtual Visor, Sensics xSight; www.vrealities.com) could solve the problem of the space on the head needed to place the fNIRS patch more correctly and comfortably. Another way might consist in the adoption of stereoscopic and auto-stereoscopic 3D displays, although it would be detrimental for VR immersion.
The use of fNIRS has resulted in considerable benefits for cognitive neuroscience studies. The procedure is non-invasive and capable of not constraining the participant to be in huge machineries such as those for PET and fMRI. It allows a natural position compared to the horizontal position assumed within the PET or fMRI. Therefore, fNIRS can allow the investigation of classic neuropsychological disorders in a way not possible previously.
The implementation of fNIRS in the analysis of the cerebral functions that subtend VR experiences represents another important goal. The exploration and the in-depth examination of different methodologies and solutions for VR applications will be the most important topic of future studies. In conclusion, fNIRS has the potentiality to discover new frontiers in brain research and VR applications.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
This research was supported by the European Project CEEDS, The Collective Experience of Emphatic Data Systems, European Integrated Project (Number: 258749; Call: ICT 2009.8.4 Human-Computer Confluence).
References
Astur, R. S., Tropp, J., Sava, S., Constable, R. T., and Markus, E. J. (2004). Sex differences and correlations in a virtual Morris water task, a virtual radial arm maze, and mental rotation. Behav. Brain Res. 151, 103–115.
Azañón, E., Longo, M. R., Soto-Faraco, S., and Haggard, P. (2010). The posterior parietal cortex remaps touch into external space. Curr. Biol. 20, 1304–1309.
Baheux, K., Yoshizawa, M., Seki, K., and Yasunobu, H. (2006). Virtual reality pencil and paper test for neglect: a protocol. Cyberpsychol. Behav. 9, 192–195.
Baumgartner, T., Valko, L., Esslen, M., and Jäncke, L. (2006). Neural correlate of spatial presence in an arousing and noninteractive virtual reality: an EEG and psychophysiology study. Cyberpsychol. Behav. 9, 30–45.
Bischof, W. F., and Boulanger, P. (2003). Spatial navigation in virtual reality environments: an EEG analysis. Cyberpsychol. Behav. 6, 487–495.
Bjoertomt, O., Cowey, A., and Walsh, V. (2001). Spatial neglect in near and far space investigated by repetitive transcranial magnetic stimulation. Brain 125, 2012–2022.
Brody, A. L., Mandelkern, M. A., Jarvik, M. E., Lee, G. S., Smith, E. C., and Huang, J. C., Bota, R. G., Bartzokis, G., and London, E. D. (2004). Differences between smokers and nonsmokers in regional gray matter volumes and densities. Biol. Psychiatry 55, 77–84.
Calhoun, V. D., Carvalho, K., Astur, R., and Pearlson, G. D. (2005). Using virtual reality to study intoxication effects on the neural correlates of simulated driving. Appl. Psychophysiol. Biofeedback 30, 285–306.
Calhoun, V. D., Pekar, J. J., and Pearlson, G. D. (2004). Alcohol intoxication effects on simulated driving: exploring alcohol-dose effects on brain activation using functional MRI. Neuropsychopharmacology 29, 2097–2107.
Campbell, Z., Zakzanis, K. K., Jovanovski, D., Joordens, S., Mraz, R., and Graham, S. J. (2009). Utilizing virtual reality to improve the ecological validity of clinical neuropsychology: an fMRI case study elucidating the neural basis of planning by comparing the Tower of London with a three-dimensional navigation task. Appl. Neuropsychol. 16, 295–306.
Clemente, M., Rey, B., Alcañniz, M., Bretón-López, J., Moragrega, I., Bañnos, R. M., Botella, C., and Avila, C. (2010). Contributions of functional magnetic resonance in the field of psychological treatments with virtual reality. Stud. Health Technol. Inform. 154, 197–201.
Combe, E., Suzuki, K., Chao, Z. C., and Fujii, N. (2010). Facing a threat in a 3D environment: evaluation of the impact of depth perception on the emotional state by skin conductance and near-infrared spectroscopy. J. Neurosci. Res. 68, e268.
Cutini, S., Scarpa, F., Scatturin, P., Jolicœur, P., Pluchino, P., Zorzi, M., and Dell'Acqua, R. (2011a). A hemodynamic correlate of lateralized visual short-term memories. Neuropsychologia 49, 1611–1621.
Cutini, S., Scatturin, P., and Zorzi, M. (2011b). A new method based on ICBM152 head surface for probe placement in multichannel fNIRS. Neuroimage 54, 919–927.
Decety, J., Perani, D., Jeannerod, M., Bettinardi, V., Tadary, B., Woods, R., Mazziotta, J. C., and Fazio, F. (1994). Mapping motor representations with positron emission tomography. Nature 371, 600–602.
Devaraj, A. (2005). Signal Processing for Functional Near-infrared Neuroimaging. Unpublished MS Thesis, Drexel University.
Di Diodato, L. M., Mraz, R., Baker, S. N., and Graham, S. J. (2007). A haptic force feedback device for virtual reality-fMRI experiments. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 570–576.
Ferber, S., and Karnath, H.-O. (2001). How to assess spatial neglect – line bisection or cancellation tasks? J. Clin. Exp. Neuropsychol. 23, 599–607.
Fink, G. R., Marshall, J. C., Shah, N. J., Weiss, P. H., Halligan, P. W., Grosse-Ruyken, M., Ziemons, K., Zilles, K., and Freund H.-J. (2000). Line bisection judgments implicate right parietal cortex and cerebellum as assessed by fMRI. Neurology 54, 1324–1331.
Fink, G. R., Marshall, J. C., Weiss, P. H., Toni, I., and Zilles, K. (2002). Task instructions influence the cognitive strategies involved in line bisection judgements: evidence from modulated neural mechanisms revealed by fMRI. Neuropsychologia 40, 119–130.
Fink, G. R., Marshall, J. C., Weiss, P. H., and Zilles, K. (2001). The neural basis of vertical and horizontal line bisection judgments: an fMRI study of normal volunteers. Neuroimage 14, 46–51.
Foxe, J. J., McCourt, M. E., and Javitt, D. C. (2003). Right hemisphere control of visuospatial attention: line-bisection judgments evaluated with high-density electrical mapping and source analysis. Neuroimage 19, 710–726.
Franceschini, M. A., Toronov, V., Filiaci, M. E., Gratton, E., and Fantini, S. (2000). On-line optical imaging of the human brain with 160-ms temporal resolution. Opt. Express 6, 49–57.
Gamberini, L., Seraglia, B., and Priftis, K. (2008). Processing of peripersonal and extrapersonal space using tools: evidence from visual line bisection in real and virtual environments. Neuropsychologia 46, 1298–1304.
Garrison, B., and Ellard, C. (2009). The connection effect in the disconnect between peripersonal and extrapersonal space. J. Vis. 9, article 79.
Gassert, R., Chapuis, D., Bleuler, H., and Burdet, E. (2008). Sensors for applications in magnetic resonance environments. IEEE/ASME Trans. Mechatron. 13, 335–344.
Hoffman, H. G., Richards, T. L., Coda, B., Bills, A. R., Blough, D., Richards, A. L., and Sharar, S. R. (2004). Modulation of thermal pain-related brain activity with virtual reality: evidence from fMRI. Neuroreport 15, 1245–1248.
Horikawa, E., Okamura, N., Tashiro, M., Sakurada, Y., Maruyama, M., Arai, H., Yamaguchi, K., Sasaki, H., Yanai, K., and Itoh, M. (2005). The neural correlates of driving performance identified using positron emission tomography. Brain Cogn. 58, 166–171.
Hoshi, Y. (2005). Functional near-infrared spectroscopy: potential and limitations in neuroimaging studies. Int. Rev. Neurobiol. 66, 237–266.
Hoshi, Y., Tsou, B. H., Billock, V. A., Tanosaki, M., Iguchi, Y., Shimada, M., Shinba, T., Yamada, Y., and Oda, I. (2003). Spatiotemporal characteristics of hemodynamic changes in the human lateral prefrontal cortex during working memory tasks. Neuroimage 20, 1493–1504.
Hribar, A., Koritnik, B., and Munih, M. (2009). Phantom haptic device upgrade for use in fMRI. Med. Biol. Eng. Comput. 47, 677–684.
Huppert, T. J., Hoge, R. D., Diamond, S. G., Franceschini, M. A., and Boas, D. A. (2006). A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans. Neuroimage 29, 368–382.
Hurwitz, M., Valadao, D., and Danckert, J. (2011). Functional MRI of dynamic judgments of spatial extent. Exp. Brain Res. 214, 61–72.
Iaria, G., Petrides, M., Dagher, A., Pike, B., and Bohbot, V. D. (2003). Cognitive strategies dependent on the hippocampus and caudate nucleus in human navigation: variability and change with practice. J. Neurosci. 23, 5945–5952.
Irani, F., Platek, S. M., Bunce, S., Ruocco, A. C., and Chute, D. (2007). Functional near-infrared spectroscopy (fNIRS): an emerging neuroimaging technology with important applications for the study of brain disorders. Clin. Neuropsychol. 21, 9–37.
Izzetoglu, K., Bunce, S., Izzetoglu, M., Onarall, B., and Pourrezaeil, K. (2003). fNIR spectroscopy as a measure of cognitive task load. Proceedings of the 25th Annual International Conference of the IEEE EMBS Cancun, Mexico. September 2003, 17–21.
Jancke, L., Brunner, B., and Esslen, M. (2008). Brain activation during fast driving in a driving simulator: the role of the lateral prefrontal cortex. Neuroreport 19, 1127–1130.
Jeong, M., Tashmo, M., Singh, L. N., Yamaguchi, K., Horikawa, E., Miyake, M., Watanuki, S., Iwata, R., Fukuda, H., Takahashi, Y., and Itort, M. (2006). Functional brain mapping of actual car-driving using [lSF]FDG-PET. Ann. Nucl. Med. 20, 623–628.
Jewell, G., and McCourt, M. E. (2000). Pseudo neglect: a review and meta-analysis of performance factors in line bisection tasks. Neuropsychologia 38, 93–110.
Lee, J. H., Lim, Y., Wiederhold, B. K., and Graham, S. J. (2005). A functional magnetic resonance imaging (fMRI) study of cue-induced smoking craving in virtual environments. Appl. Psychophysiol. Biofeedback 30, 195–204.
Lee, S. H., Han, D. H., Oh, S., Lyoo, I. K., Lee, Y. S., Renshaw, P. F., and Lukas, S. E. (2009). Quantitative electroencephalographic (qEEG) correlates of craving during virtual reality therapy in alcohol-dependent patients. Pharmacol. Biochem. Behav. 91, 393–397.
Li, Z., Zhang, M., Zhang, X., Dai, S., Yu, X., and Wang, Y. (2009). Assessment of cerebral oxygenation during prolonged simulated driving using near-infrared spectroscopy: its implications for fatigue development. Eur. J. Appl. Physiol. 107, 281–287.
Longo, M. R., and Lourenco, S. F. (2006). On the nature of near space: effects of tool use and the transition to far space. Neuropsychologia 44, 977–981.
Maguire, E. A., Burgess, N., Donnett, J. G., Frackowiak, R. S. J., Frith, C. D., and O'Keefe J. (1998). Knowing where and getting there: a human navigation network. Science 280, 921–924.
Meda, S. A., Calhoun, V. D., Astur, R. S., Turner, B. M., Ruopp, K., and Pearlson, G. D. (2009). Alcohol dose effects on brain circuits during simulated driving: an fMRI study. Hum. Brain Mapp. 30, 1257–1270.
Merians, A. S., Poizner, H., Boian, R., Burdea, G., and Adamovich, S. (2006). Sensorimotor training in a virtual reality environment: does it improve functional recovery poststroke? Neurorehabil. Neural Repair 20, 252–267.
Miyai, I., Tanabe, H. C., Sase, I., Eda, H., Oda, I., Konishi, I., Tsunazawa, Y., Suzuki, T., Yanagida, T., and Kubota, K. (2001). Cortical mapping of gait in humans: a near-infrared spectroscopic topography study. Neuroimage 14, 1186–1192.
Niedermeyer, E., and Lopes da Silva, F. H. (2004). Electroencephalography: basic principles, clinical applications, and related fields. New York: Lippincott Williams and Wilkins
Optale, G., Urgesi, C., Busato, V., Marin, S., Piron, L., Priftis, K., Gamberini, L., and Capodieci, S. (2010). Controlling memory impairment in the elderly using virtual reality memory training: a randomized controlled, pilot study. Neurorehabil. Neural Repair 24, 348–357.
Owen-Reece, H., Smith, M., Elwell, C. E., and Goldstone, J. C. (1999). Near-infrared spectroscopy. Br. J. Anaesth. 82, 418–426.
Parslow, D. M., Morris, R. G., Fleminger, S., Rahman, Q., Abrahams, S., and Recce, M. (2005). Allocentric spatial memory in humans with hippocampal lesions. Acta Psychol. (Amst). 118, 123–147.
Peers, P. V., Ludwig, C. J. H., Rorden, C., Cusack, R., Bonfiglioli, C., Bundesen, C., Driver, J., Antoun, N., and Duncan, J. (2005). Attentional functions of parietal and frontal cortex. Cereb. Cortex 15, 1469–1484.
Perani, D., Fazio, F., Borghese, N. A., Tettamanti, M., Ferrari, S., Decety, J., and Gilardi, M. C. (2001). Different brain correlates for watching real and virtual hand actions. Neuroimage 14, 749–758.
Pisella, L., Alahyane, N., Blangero, A., Thery, F., Blanc, S., and Pelisson, D. (2011). Right-hemispheric dominance for visual remapping in humans. Philos. Trans. R. Soc. Lond. B Biol. Sci. 366, 572–585.
Pugnetti, L., Mendozzi, L., Barbieri, E., Rose, F. D., and Attree, E. A. (1996). “Nervous system correlates of virtual reality experience,” in Proceedings of 1st European Conference on Disability, Virtual Reality and Associated Technologies, Maidenhead, UK, 1996, 239–246.
Rorden, C., Bergerb, M. F., and Karnath, H.-O. (2006). Disturbed line bisection is associated with posterior brain lesions. Brain Res. 1080, 17–25.
Savitzky, A., and Golay, M. J. E. (1964). Smoothing and differentiation of data by simplified least squares procedures. Anal. Chem. 36, 1627–1639.
Schroeter, M. L., Bucheler, M. M., Muller, K., Uludag, K., Obrig, H., Lohmann, G., Tittgemeyer, M., Villringer, A., and Cramon, D. Y. (2004). Towards a standard analysis for functional near-infrared imaging. Neuroimage 21, 283–290.
Schroeter, M. L., Zysset, S., Kupka, T., Kruggel, F., and Cramon, D. Y. (2002). Near-infrared spectroscopy can detect brain activity during a color-word matching Stroop task in an event-related design. Hum. Brain Mapp. 17, 61–71.
Sevick, E. M., Chance, B., Leigh, J., Nioka, S., and Maris, M. (1991). Quantitation of time- and frequency-resolved optical spectra for the determination of tissue oxygenation. Anal. Biochem. 195, 330–351.
Shimada, S., Hiraki, K., Matsuda, G., and Oda, I. (2004). Decrease in prefrontal hemoglobin oxygenation during reaching tasks with delayed visual feedback: a near-infrared spectroscopy study. Brain Res. Cogn. Brain Res. 20, 480–490.
Slobounov, S., Wu, T., Hallett, M., Shibasaki, H., Slobounov, E., and Newell, K. (2006). Neural underpinnings to postural responses to visual field motion. Biol. Psychol. 72, 188–197.
Strangman, G., Boas, D. A., and Sutton, J. P. (2002). Non-invasive neuroimaging using near-infrared light. Biol. Psychiatry 52, 679–693.
Taga, G., Asakawa, K., Maki, A., Konishi, Y., and Koizumi, H. (2003). Brain imaging in awake infants by near-infrared optical topography. Proc. Natl. Acad. Sci. U.S.A. 100, 10722–10727.
Takahashi, T., Takikawa, Y., Kawagoe, R., Shibuya, S., Iwano, T., and Kitazawa, S. (2011). Influence of skin blood flow on near-infrared spectroscopy signals measured on the forehead during a verbal fluency task. Neuroimage 57, 991–1002.
Takeuchi, Y. (2000). Change in blood volume in the brain during a simulated aircraft landing task. J. Occup. Health 42, 60–65.
Tanaka, T., Ifukube, T., Sugihara, S., and Izumi, T. (2010). A case study of new assessment and training of unilateral spatial neglect in stroke patients: effect of visual image transformation and visual stimulation by using a head mounted display system (HMD). J. Neuroeng. Rehabil. 7, 20.
Ter-Pogossian, M. M., Phelps, M. E., Hoffman, E. J., and Mullani, N. A. (1975). A positron-emission transaxial tomograph for nuclear imaging (PET). Radiology 114, 89–98.
Thiel, C. M., Zilles, K., and Fink, G. R. (2004). Cerebral correlates of alerting, orienting and reorienting of visuospatial attention: an event-related fMRI study. Neuroimage 21, 318–328.
Tomikawa, M., Hong, J., Shiotani, S., Tokunaga, E., Konishi, K., Ieiri, S., Tanoue, K., Akahoshi, T., Maehara, Y., and Hashizume, M. (2010). Real-time 3-dimensional virtual reality navigation system with open MRI for breast-conserving surgery. J. Am. Coll. Surg. 210, 927–933.
Villringer, A., and Chance, B. (1997). Non-invasive optical spectroscopy and imaging of human brain function. Trends Neurosci. 20, 435–442.
Villringer, A., and Dirnagl, U. (1995). Coupling of brain activity and cerebral blood flow: basis of functional neuroimaging. Cerebrovasc. Brain Metab. Rev. 7, 240–276.
Waberski, T. D., Gobbele, R., Lamberty, K., Buchner, H., Marshall, J. C., and Fink, G. R. (2008). Timing of visuo-spatial information processing: electrical source imaging related to line bisection judgements. Neuropsychologia 46, 1201–1210.
Weiss, P. H., Marshall, J. C., Wunderlich, G., Tellmann, L., Halligan, P. W., Freund, H. J., Zilles, K., and Fink, G. R. (2000). Neural consequences of acting in near versus far space: a physiological basis for clinical dissociations. Brain 123, 2531–2541.
Keywords: virtual reality, fNIRS, line bisection, attention, brain imaging, pseudoneglect
Citation: Seraglia B, Gamberini L, Priftis K, Scatturin P, Martinelli M and Cutini S (2011) An exploratory fNIRS study with immersive virtual reality: a new method for technical implementation. Front. Hum. Neurosci. 5:176. doi: 10.3389/fnhum.2011.00176
Received: 26 September 2011; Accepted: 14 December 2011;
Published online: 28 December 2011.
Edited by:
Shuhei Yamaguchi, Shimane University, JapanReviewed by:
Naoyuki Osaka, Kyoto University, JapanHiroshi Nittono, Hiroshima University, Japan
Emmanuelle Combe, RIKEN, Japan
Copyright: © 2011 Seraglia, Gamberini, Priftis, Scatturin, Martinelli and Cutini. This is an open-access article distributed under the terms of the Creative Commons Attribution Non Commercial License, which permits non-commercial use, distribution, and reproduction in other forums, provided the original authors and source are credited.
*Correspondence: Luciano Gamberini, Department of General Psychology, University of Padua, Via Venezia 12, 35131, Padova (PD), Italy. e-mail: luciano.gamberini@unipd.it