- 1Escuela Profesional de Medicina Humana, Universidad Privada San Juan Bautista (UPSJB), Lima, Peru
- 2Facultad de Ingeniería y Arquitectura, Universidad de Lima, Lima, Peru
- 3Universidad de Ingeniería y Tecnología (UTEC), Lima, Peru
The introduction of Augmented Reality (AR) has attracted several developments, although the people’s experience of AR has not been clearly studied or contrasted with the human experience in 2D and 3D environments. Here, the directional task was applied in 2D, 3D, and AR using simplified stimulus in video games to determine whether there is a difference in human answer reaction time prediction using context stimulus. Testing of the directional task adapted was also done.
Research question: Are the main differences between 2D, 3D, and AR able to be predicted using Markov chains?
Methods: A computer was fitted with a digital acquisition card in order to record, test and validate the reaction time (RT) of participants attached to the arranged RT for the theory of Markov chain probability. A Markov chain analysis was performed on the participants’ data. Subsequently, the way certain factors influenced participants RT amongst the three tasks time on the accuracy of the participants was sought in the three tasks (environments) were statistically tested using ANOVA.
Results: Markov chains of order 1 and 2 successfully reproduced the average reaction time by participants in 3D and AR tasks, having only 2D tasks with the variance predicted with the current state. Moreover, a clear explanation of delayed RT in every environment was done. Mood and coffee did not show significant differences in RTs on a simplified videogame. Gender differences were found in 3D, where endogenous directional goals are in 3D, but no gender differences appeared in AR where exogenous AR buttons can explain the larger RT that compensate for the gender difference. Our results suggest that unconscious preparation of selective choices is not restricted to current motor preparation. Instead, decisions in different environments and gender evolve from the dynamics of preceding cognitive activity can fit and improve neurocomputational models.
Introduction
In the modern world, 3D stimuli is now commonplace, with most people having experienced it through education or entertainment in the form of 3D TV, as well as 3D and 2D stimuli on 2D screens. More recently, augmented reality (AR) has become more common, now being one of the main ways to receive information and in videogame controlling (e.g., DeMarle, 2017). The AR of incoming stimulus has a role in memory and cognitive load, but how AR changes memory is not clear (Schneider et al., 2020). In this study, how attention and the previous response manage AR were addressed by means of stimulus in 2D, 3D, and AR environments.
The Posner paradigm of Posner permits the cost and benefit of reaction times (RTs) to be quantified (Posner, 1980), in the order of 80% valid goal stimulus in events, stimulus was presented in the area of the screen indicated by a warning signal (arrow). In 1984, this paradigm was used to study the RTs of people with damage to the parietal cortex with a greater difference observed in contralateral and ipsilateral in right parietal patients (Posner et al., 1984). Later, Posner’s paradigm was used to study task responses via ocular saccadic movement or by the pressing of a keyboard button as seen in Figure 1. The Posner paradigm has also demonstrated that microsaccades can be used to map visual orientation. This is due to the amplitude of the saccadic movement and showed RTs being higher for disabled stimuli compared to neutral stimulus or even children (Engbert and Kliegl, 2003). Moreover, Posner and colleagues have shown neuroimaging differences according to the RTs in the Attention Network Task (ANT; Fan et al., 2005) where alerting was found mainly on fronto-parietal and thalamic activation. Orienting has also been found in the parietal areas and in the frontal eye fields (FEF) and executive control has been shown to activate the Anterior Cingulate Cortex (ACC) along with other areas. An auditory version of this paradigm found a similar and complementary activation of frontal areas changing with the difference between conditions and the previous condition as well (Mugruza-Vassallo et al., 2021). This means, the warning signal in visual studies has shown real outcomes from the analysis done by cognitive human computation. The present research removed this possible covariate with a simplified version of an arrow in a videogame in 2D, 3D, and AR.
Figure 1. Experimental screens similar to Posner et al. (1984); Torres-Tejada et al. (2020); and Mugruza-Vassallo and Potter (2019). Those works allow us to make a simplified video-game experiment and also to ask: Are 2D, 3D, and augmented reality (AR) main differences processed able to predict with Markov chains?
Recently, Go-Go experiments and variants of Posner’s paradigm such as ANT, coupled with tested memory, found that there was not necessarily any change in attention after the first episode of mania. However, it is possible to change other cognitive measures such as IQ, processing speed, memory work, verbal fluency, and certain executive functions (except for Go/No-Go tasks) with p < 0.05 (Daglas et al., 2017). Menozzi et al. (2007) tested AR stimuli of 200, 300, and 600 ms duration to locate the visual stimulus “3” in a sequence of numbers and interpreted this to mean that localisation depends on the ability to focus attention to one stimulus in an undetermined time after the signal or arrow was sent (Posner, 1980). Moreover, at the time of drawing up this manuscript a patent application has been made for an invention that relies on the benefit of the reaction time of Posner’s paradigm (1980) to improve the virtual reality teaching environment using warning signals for the adjustment of the screen, which has the advantage of attracting the attention of the user (Shahal, 2018). Alternatively, Torres-Tejada et al. (2020) developed videogames and showed for five types of visual stimuli (four types of Go-Go events, see adaption of their screens in top left part of graphical abstract). They found that the angle of presentation and the relative size of the stimulus were significant for computations between 2D and 3D. Torres-Tejada’s videogames used arrows as a guide in a similar way to the Posner paradigm. In this work, the hypothesis posed asks: is it possible to simplify the number of stimuli in a videogame and does the Go-Go task allow you to study the difference between 2D, 3D, and AR stimuli?
Therefore, in the present research, AR was compared to 2D and 3D to assess selective attention by way of reaction times (e.g., around 1,000 ms) stimuli whereas the original version of the paradigm of Posner places stimuli of alert by arrow (called endogenous signals) prior to the presentation of goal signals.
Regarding the technical part, some AR trials have used a combination of AR Tool Kit, Visual Studio C++, 3D Max, and Adobe Illustrator tools (Chen et al., 2015). In this work the previous use of Unity 3D and Vuforia made by Torres-Tejada et al. (2020) was extended from 2D and 3D stimuli to AR stimuli. Therefore, part of the technical challenge is to achieve a reasonable resolution time in data acquisition that allows measurement of the reaction times in UNITY 3D.
Taraghi and colleagues studied the dependence of the immediately preceding answer in a sequence of questions with Markov chains. The response data was acquired through an application called “1 × 1 trainer,” a tool designed for elementary school children to learn multiplication. After analysis of response times, a Markov chain of order five confirmed the probability of response on 442,910 questions answered by 3,381 children over 11,711 sessions. Some children did not take the questions seriously and other children left before the test was over (Taraghi et al., 2014). Consequently, the present work considered adult participants and therefore, the Markov chains were expected to be of an order less than five across the different presentations of 2D, 3D, and AR.
The formula (1) was used to calculate the different types of responses, where P is the probability and X the random variables were defined according to the expression (1) given as:
We assume that the probabilities do not change as a function of time, hence the Markov chain transitions are time-homogeneous (Busemeyer and Townsend, 1992). Therefore, a Markov chain of order k is described formally as (2) used by Taraghi et al. (2014).
Behavioural models of goal stimuli have been used as Markov decision models (Savalia et al., 2016) or even response models for linguistic or musical events (Forth et al., 2016). Limitations in both models were that they did not consider cases of similar tasks in different modalities, moreover not citing or giving examples of cognitive responses in those models. In the present experiment and interpretation, we seek to indicate to what extent the Markov chain fits the response to events in 2D, 3D, and AR.
Conversely, the data analysis of RTs is commonly used in the Analysis of Variance (ANOVA) of the data (e.g., Potter et al., 2001; Mugruza-Vassallo, 2016a). Furthermore, other recording methods, for visual cues with letters along with results in functional Magnetic Resonance Imaging (fMRI) by Koechlin et al. (2003), Electroencephalography (EEG; Mugruza-Vassallo and Potter, 2019) and auditory fMRI results (Mugruza-Vassallo et al., 2021) have suggested that the context of the immediately preceding stimulus is significant and modulate the measurements of cognitive outcomes by recruiting a larger brain area when motor response changes between action and no-action, depending of previous novel response as a kind of Bayes analysis. Results from preceding stimuli were observed in the lower part of Figure 1, where the EEG dependence was found in the correlation of the properties of the current and the immediately prior stimuli. Moreover, prior research has found no evidence of any deficiency in N100 type potentials (Mugruza-Vassallo, 2016b; Mugruza-Vassallo and Potter, 2019) in agreement with Rosburg (2018). Likewise, the research of Samrani et al. (2018) also involved the context of working memory relating to stimuli. The previous research considered the dependence of the previous answer taken here as a possible Markov process, the present research considers a subset of characteristics to simplify the videogame compared with the case of Torres-Tejada et al. (2020), where five types of stimuli were used (see upper part of Figure 1). The present work aims to study the dependence of previous stimulus using Markov chains using only few types of stimuli in graphic format. Therefore, here it was postulated that the selective attention can be studied with responses based on an experiment which add and simplifies works of: (a) top Experiment 1: (Engbert and Kliegl, 2003), which timed three different screens (a fixing cross, an alert signal with a right arrow and a target to which the participants responded); (b) middle: 2D and 3D experiments showing RTs results altered due to stimulus properties (viewpoint and relative size); and (c) bottom: Auditory sequence with several types of stimulus has shown a strong dependence of sound properties and sequence.
Bearing in mind all the literature, the research question aimed to answer whether the Markov chains could explain occurrence states for 2D, 3D, and AR environments. Moreover, the hypothesis states that the Markov chain method in the experiment can predict the variance in those environments (2D, 3D, and AR) using the RT data. Therefore, the aim was to identify which of the 2D, 3D, and AR environments showed better attention outcomes to a simplified visual decision task by measuring accuracy and RTs, and then using Markovian reasoning to decipher the results. The question was addressed by adopting a Go-Go-type experiment which displayed an arrow that pointed either left or right to simplify the videogame stimuli.
Material and Methods
Participants
The experiment was carried out using 22 participants (six women) in the morning, aged from 18 to 24 years old (mean = 21.3 standard deviation = 1.1 years). Participants signed an informed consent according to the University ethical rules. Three selection criteria were used to choose participants: motivation to take the test, a general experience of playing videogames and declared that they had no impairments, being under pressure or stress that might affect their ability to pay due attention to the videogame.
Experiment
The software used to develop the videogame and acquire RT data was Unity 3d software (2004), a high level graphics engine that allowed us to develop three videogames for different platforms through an editor and a scripting language. To make the AR video, Vuforia complement was used (Vuforia, 2012), a useful tool to build applications based on augmented reality (e.g., Imbert et al., 2013). This is a widely used complement of software, used for computer games and mobile. The development of the present experiment’s application for 2D, 3D, and AR environments respectively, is shown in Figures 2A–C, where the same moving caricature is presented at three different environments.
Figure 2. Simplified videogame using arrows on (A) 2D, (B) 3D, and (C) AR. The arrow was either to the left or to the right.
In the experiment 120 stimuli were presented per block (360 in total). Each event at each block game started by displaying the arrow pointing either left or right for an average of 1 s, as shown in Figure 3. A random and variable time of between 0.7 s and 1.3 s was used to display the next stimulus. The participant had to identify which way the arrow was pointing and press the button to his right or left, as appropriate. Each participant was sitting down in front of a desk and maintaining head position at a viewing distance of approximately 60 cm.
Data acquisition
RT in milliseconds (ms) were recorded, RT data were extracted whenever the user physically pressed the keyboard buttons (2D, 3D) or the virtual button (AR with the support of Logitech 922c videocamera and screen), according to the video game. RTs were measured from the moment when the arrow appeared until the virtual/physical button was pressed. Recorded data were based on the keys pressed, the participant’s RT and whether or not the correct key had been pressed, which was recorded as “true” (V) or “false” (F) state.
Data analysis
All RTs less than 200 ms were filtered out because according to Welford (1988) this is the amount of time taken for a person to receive and process a signal. Therefore, the stimuli detection processing time is less than 200 ms, being around 170 ms the case of the human face (e.g., Rousselet et al., 2008). Therefore, the probabilities of the various RTs of response types were analysed throughout the experiment using the Markov statistical method. Finally, RTs obtained when going from one state n to another state n + 1 according to the Markov chain were analysed in detail.
Following Equation (2), the first order Markov chain (k = 1) and second order (k = 2) were described.
A Markov chain of order k = 1 is described formally as follows:
A Markov chain of order k = 2 is described formally as follows:
The states, through which the participants were: right answer at state V (correct) and wrong answer at state F (wrong). The Markov chain used for three consecutive stimuli is seen in Figure 3.
Markov chain for reaction times
To obtain an order 3 Markov chain from the recovered data, a State Matrix was assembled. The recovered data was then coded: “1” representing correct answers and “0” representing incorrect answers. As a result, we have a matrix of states, as represented in Table 1. This allows a quantification of the occurrences for each Markov chain and to completely build the Markov chain.
In the Chain of States corresponding values were set at 0’s and 1’s for stimulus correctly or incorrectly answered. Each row is equivalent to three consecutive stimuli in respect to the 120 used per each participant. For example, the first chain of stimulus 1–3, second chain, 2–4, and so on.
Finally, we converted the 0’s and 1’s to base 10 and added each chain of states formed. In analysing the three consecutive states from the 120 stimuli collected, we obtained 118 chains of states per game block made. These data represented the information needed for the final state and from this we begin to back up, in order to obtain the probabilities.
The same process was carried out for each block and for each participant and a matrix of all the videogames was assembled and afterwards a Markov chain was made per each video game.
ANOVA of prior state RTs
The normality of the RTs was evaluated using the Shapiro-Wilk test of normality. It was found that the assumption of normality was valid. Also each RTs distribution was plotted in a histogram, no bimodal distribution was found. In order to determine whether the previous state had an influence on the current state, an ANOVA analysis was carried out on bases of the RTs of the previous states (Markov chain) and the visual environments (i.e., 2D, 3D, and AR).
Variables that might have had an effect RT, namely the influence of coffee consumption, mood and gender were also analysed with ANOVA.
Results
Markov chain results
In order to consolidate the data, a Markov chain was created for each video game containing the data of all the participants, as they are presented in Figures 4A–C, reflecting their true (V) and false (F) answers.
Figure 4. States in the AR game for all participants (A) in the 2D game f. (B) In the 3D game. (C) In the AR game.
Figure 4A represents the Markov chains formed for the 2D. If the initial state is “FF” then probabilities have increased for the current state “F” (relatively with respect to the expected 12.5%). This means that if the person fails twice consecutively, the next state has a 25% chance (double the expected) of getting it wrongly and 75% of performing it correctly. If the initial state is “FV,” the probability has changed to 96.2% for participants performing the experiment correctly at the current state. If the case is “VF” or “VV,” probability of failure is very low and a large part of the probability that the answer is in the “true” state.
With respect to the Markov chain representing the 3D game (illustrated in Figure 4B), when a previous state of “FF” or “FV” was presented, there is zero probability of performing the following stimulus erroneously and instead, 100% probability of performing it correctly. In the case of “VF” and “VV,” very low probabilities were perceived, similar to what was seen in the 2D video game chain.
Finally, in Figure 4C, we recorded a large variation in probabilities. Evidently, the probability of error is greater than that in the 2D and 3D Markov chains; however, if the previous state was “VV,” the evidence suggests that there is a large probability of correctly performing the next stimulus. Additionally, if the previous state is “F” in the AR, the probability of performing the subsequent stimulus well is 76.9%, which is, however, less than the probability shown in Figure 4B (2D at 96.3%) and Figure 4C (3D at 98.7%).
Prior state influenced RTs
Table 2 shows the ANOVA results for the participant response states (V: true, F: false) at different environments (2D, 3D, and AR). Three cases were analysed: a Markov model that considers only the current state, a Markov model that considers the current and previous state, and a Markov model which considers the current state and two previous states. In any of these models, the current state is level 0, the previous state is level 1, and so on.
In Table 2; the 0–1 levels show an influence on RTs in the two-state model. This was also seen in the three-state model, with levels of 0–2. Previous states had a significant influence (p < 0.05) in the RT of the current state.
Coffee, mood, and gender in RTs
The analysis also considered the effect of other cofactors that might have affected the true or false responses of the participants. These were the effects of coffee consumption, mood, and gender on RTs were analysed using ANOVA.
No evidence was found that coffee consumption or mood were correlated with RT (see Table 3). However, coffee consumption is a possible study limitation that was not controlled. Previous experiments had considered excluding participants who drink more than 10 cups of coffee or had consumed coffee 3 h prior to the experiment (e.g., de Bruin et al., 2004), although other studies deprived coffee consumption (Chen et al., 2021). Furthermore the present study did not consider consecutive days of coffee administration as other studies on different coffee administrations suggested (Judelson et al., 2005) and we also did not make a comparison based on healthy and vulnerable populations with cardiovascular or sleep impairment (Temple et al., 2017).
Conversely, the 3D game introduced a different range of significant characteristics in RT between men and women (p = 0.003 @ F = 10.475). This result agreed time was interpreted shorter by women at different visual spatial angles (of 25°) in a modified Posner cueing paradigm (Cooney et al., 2017). More interesting, errors in True or False answers were found to be significant in 2D (p = 0.02 @ F = 5.8553). Supporting these results, it is believed motor control compensated for gender on the AR experiment.
Finally, bearing in mind gaming as a meaningful and purposeful activity for gamers, and as an extension of the present work, it may be useful to consider including depth mood evaluation questions, as in Shi et al. (2019) who recently found there to be caused by participants understanding phrases, in different meaning tasks (trait implying vs. control phrases).
Improving Markovian brain modelling
Considering overall results and cognitive science application, the different 2D, 3D, AR environments (p3) have resulted in changes to motor control (p2) depending on the current motor response (m(k)) and the immediately prior motor response (m(k − 1)), see neurocomputational model in Figure 5, extending results of Khalvati et al. (2019), Mugruza-Vassallo et al. (2021), and according to auditory results of Mugruza-Vassallo and Potter (2019). RT distribution was almost uniform across participants. Therefore those “no changes within the blocks” should be related to the environment and not necessarily to the “brain” selector inside p1 (e.g., repetitive or non-repetitive), leaving e (k − 1) outside p1 (black box). A future experiment would be to change 2D, 3D, and AR in the same block to test how “brain” selector could change environment control.
Modelling limitations
The study did not use brain imaging techniques, leaving the rest of the control as a black box (p1), namely some combination of eye-tracking, EEG, and fMRI to best explain the modulation of the present findings. For example, recently it has been found that the number of prefrontal areas used during an auditory decision task was modulated by motor and inhibition response (Mugruza-Vassallo et al., 2021). Therefore, the task were mainly conducted on attention and decision-making without taking into account multitasking, leaving p3 without evidence of multimodal or multitasking distribution.
Discussion
Markov chains explain part of the variance in 2D, 3D, and AR
The differences observed with 2D using a Level 0 in Markov response, 3D using Level 1 and AR using Level 2 draws attention to the complexity of human reaction behaviour, i.e., RTs on accuracy classification in a visual paradigm, which is related to motor findings on auditory processing found in fMRI (Mugruza-Vassallo et al., 2021). To the knowledge of the present authors, the paradigm of Posner is the first and most common paradigm used when attempting to relate to RT in the orienting of attention paradigms. The differences observed in 2D, 3D, and AR in this study could be seen as an extension of recent neuroimaging studies, which have discussed auditory stimulus properties along with the orienting of attention, whilst considering the immediately prior stimulus (using EEG in Mugruza-Vassallo and Potter, 2019; using fMRI in Mugruza-Vassallo et al., 2021), using different regressors than visual modality (Mugruza-Vassallo, 2016b) and how visual properties can guide 2D and 3D RTs attention environments (Torres-Tejada et al., 2020). Moreover, the present authors are unaware of any articles that have sought to apply the direct choice paradigm to AR and use Markov to study cognitive human computation.
Additionally, the implications of these results may help to improve the understanding of memory in AR attentional tasks. Bearing in mind Schneider’s work (2020), it would help to develop chip devices to boost memory while working between 3D and AR. For example in Alzheimer brain processing, bearing in mind that ageing attenuates value sensitivity and integration during decision-making and requires more environmental compensation than younger adults (Chen et al., 2021) as well as extending studies of habits in executive function (Obeso et al., 2021).
Prior state influenced RTs
The response of selective attention by means of contextual memory of a previous measure used Markov chains. The Markov chain response patterns were observed to improve when people made errors in their response to the stimuli, which resulted in RT improvement (without feedback) and dependent on the previous state. These results and the different RTs Gaussian distributions, see Table 3 for statistical means and standard deviation are consistent with different visual cognition systems as of Batallones’ findings (Batallones et al., 2015). Indeed Gaussian RTs distributions were found similar in 2D, 3D, and AR but with clearly different average RTs for AR. Batallones’ results considered the RT, however, the results of this study extend the idea to the immediately prior context of the experience affecting attentional states. Attention was studied with P300 according to the analyses in auditory cases seen with EEG (Mugruza-Vassallo and Potter, 2019) and visual in the posterior part to the pre-motor cortex seen with fMRI (Koechlin et al., 2003). Even more, the results presented here of prior state influencing RTs would be an extension to the visual environments of the anticipatory model seen in linguistics and music (Forth et al., 2016). Moreover, an extension of the analysis could be used to test for cognitive bias, for example, to avoid repetitions or random human recognition with Markov Chains (Baena-Mirabete et al., 2020). Also, our results follow and specify previous results (Soon et al., 2013), suggesting that this unconscious preparation of selective choices is not restricted to current left or right motor answer preparation. Instead, decisions in different environments (2D, 3D or AR) evolve from the dynamics of preceding cognitive activity, as shown by different Markov chains.
Alternatively, in the video game AR, the increased success when the time passes within the blocks was in line with the learning curves of the tasks (e.g., experimentally shown by Mugruza-Vassallo and Potter, 2019). This consistency is clearly related to learning the game, as the videogame is a new task the participants had to adapt to. This interpretation is may be related to a systematisation of the data, and could lead to a more precise prediction of the participants’ responses or to factors that could modulate said precision (Mugruza-Vassallo, 2016a). Improvement of prediction could allow the Go-Go attention model to be extended based on Markov decision processes that use feedback (Savalia et al., 2016). This could be achieved in three ways:
a. Adding a learning constant to the modified Bellman equation, thus adding an additional term of learning (μ),
(b) Considering a reward for the previous action R (n − 1), in which case μ = γ2. That is similar to the truncated n-step method proposed by Watkins, 1989, page 92). This method has recently been reconsidered in Mnih et al. (2016) and Barth-Maron et al. (2018).
(c) Reinforce Learning assumptions for the Markov decision process. Maybe the change would be the definition of the rule, which is more related to Equation 2 in a way the classic version is , the new version would be . Then, re-estimate the Bellman equation.
Markovian brain modelling on multitasking and therapy
Under the assumption of multitask reinforcement, a learning process built by two states in a Markovian tree, Tomov and colleagues provided an example of a participant going to a fast-food shop or coffee shop is a good way to gain his reward when hungry or weak or a diner shop if the participant is both hungry and weak. Moreover, the participant might find that the coffee shop has a better atmosphere or environment (Tomov et al., 2021), as here tested in a different way in 2D, 3D, and AR environments.
Figure 5 also considers some Bayesian analysis, where information on the choice of priors that best fit Markov chains, having a common path with Tomov et al. (2021). However, the present experiment was built by two states in a Markovian tree depending on the environment (i.e., 2D, 3D, and AR) with a single decision left or right. A more applied example may be when the participant has to choose the effective way to preserve his life as a reward in an earthquake, where the 2D, 3D or AR environments have equivalent but different warning signals. Therefore, warning signals on different environments might be better studied for this case to choose the route for evacuation, being the viable option about safety and the “brain” selector interaction with the environment should be further studied in the earthquake warning signals case.
An additional highlight was to corroborate how the previous response (correct or incorrect) influenced the RT to simplify the stimulus made on the experiment conducted here which: (a) complement the auditory task of Mugruza-Vassallo and Potter (2019) and 2D task of Koechlin et al. (2003); and (b) there is no bias with the type of 3D stimulus as found by Torres-Tejada et al. (2020), only for the responses presented to the prior stimulus. In this way, given that Torres-Tejada’s paradigm was simplified, it has allowed the analysis of three different scenarios, in a parallel way to auditory signals (e.g., Mugruza-Vassallo, 2016a; Mugruza-Vassallo and Potter, 2019). Moreover, when information gathered from electrodes are added to the analysis, motor hand responses produced when grasping objects and avoiding obstacles (Sun et al., 2018) would reveal different neural pathways (Mugruza-Vassallo et al., 2021). Furthermore, memory control mechanisms and their underlying brain networks could be studied together with these neuroimaging techniques to answer questions relating to resilience (such as in earthquakes) following event or object detection given by warning signals. For example, recent studies have focused on improving understanding of clinical responses and variation in response following trauma (Mary et al., 2020).
In addition, the results of the order of execution of the different environments in previous blocks (3D and AR) in the video games affected significantly for the 2D game. That is to say, the changes in stimuli were significant to the RT sequences of the 2D events which together with Bellman equation open the possibility for more experiments with other measures (eye-tracking, EEG, fMRI) and gaining a better understanding of neuronal path understanding.
3D and AR on gender and videogame applications
Regarding the variables of coffee, mood, and gender, gender of the participants affected the RT when the stimulus was in 3D (p = 0.003 @ F = 10.475). The foregoing is consistent with the Posner paradigm study showing how women perform endogenous visual processing tasks faster than men before turning the head 25° when the eye worked in the visual 2D environment (Cooney et al., 2017). In this way, when the stimulus presented to the right or left was evaluated and which showed significant differences between 2D, 3D, and AR, showing as a simplified response of said effects in comparison to audio-visual experiments (Córdova Berríos et al., 2018) or audio-visual including agents (Hmamouche et al., 2020).
Moreover, in AR, gender differences were not observed, meaning that exogenous AR buttons were not affected by gender. Therefore, this result is also an extension of Mitsuda et al. (2019), where exogenous processing was found different to endogenous processing using Posner’s task. Bearing in mind that during the last decade in the AR-game environment and in a therapy, the Microsoft HoloLens emulators and Unity 3D implemented a playtest environment and designed a special test environment for recent systems (Alqithami et al., 2019; Ibrahim et al., 2021). Specifically, a recent proposal for ADHD patients improved design with the measurement of the performance index (considering factors: response, impulsivity, inattention, engagement and errors throughout time slots; Alqithami, 2021). These studies would be improved with the present results bearing in mind the two previous trials on Markov chain and gender (3D vs. AR view) would improve enhancement analysis of patients’ behaviours including exogenous and endogenous interaction.
This can be interpreted as a clear difference to the bottom-up and top-down 2D mechanism interaction with the 3D and AR modalities, which would be expected in cross-modal tasks (e.g., Parmentier et al., 2010; Mayas et al., 2014; Vasilev et al., 2019; Mühlberg and Müller, 2020).
The present work should be extended, for example, to the Mario Bros video games or even comparisons between 2D and 3D (e.g., Torres-Tejada et al., 2020) extending to AR: when a third person (as stimulus) in the visual task is reached for re-identification when a participant surveys with different camera angle views (e.g., Zheng et al., 2012). An extension of the re-identification study is to understand computation of human body parts, e.g., body local representations corresponding to local similarities (Wu et al., 2018), a proposal of coattention based comparison to find best image parts correlated (Wu et al., 2021), and the difficulties in producing good feature representations (Kodirov et al., 2016) using eye-tracker could focus not only on improvements in 2D but also in 3D and AR. Therefore, interpretations could be made from cognitive human computation that other stimulus simplified was done in representation learning having the participant similar identification with all three scenarios (2D, 3D, and AR).
As an extension of videogame controlling could be explored as referred to by DeMarle (2017) and finally, the videogame reasoning here could be applied to neurogame controlling in: (a) multimodal analysis that could be extended to Cz channel laplacian (Leeb et al., 2013); or (b) machine learning processing and analysis because Petri nets use around 5 Hz steady state visual evoked potential (SSVEP) paradigm based on EEG (Sun et al., 2018).
Conclusion
The Markov chain has confirmed the hypothesis that RTs can be predicted as a result (outputs of the states) of visual stimuli in 2D with current stimulus, 3D with current and previous stimuli, and AR with current and two previous stimuli, by means contextual memory of a previous measure for the response of selective attention in a very simplified computer game experiment designed for doing some neurophysiological recording as well.
The present study did not show significant differences in hand, mood, and coffee on a simplified videogame. On the other hand, the present study found the 3D efficacy of women was different to communicate the direction of the arrow’s attention. This extends the ongoing debate about the idea that exogenous cues affect endogenous responses in a simplified directional task. Therefore, the present results provided an insight on how exogenous AR buttons generate a different type of cueing different from 3D seen on Posner’s paradigm.
Data Availability Statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://github.com/cmugruza/2D_3D_AR_rt.
Ethics Statement
The studies involving human participants were reviewed and approved by Universidad de Lima. The patients/participants provided their written informed consent to participate in this study.
Author Contributions
CM-V made the substantial contribution to the conception of the work and time calibration, i.e., proposed the idea of removing warning signals to seek for differences on AR, made part of data collection and children participant’s test, made the final analysis and main interpretation of data for the work, finally drafted the work and revised it critically for most of the intellectual content, made final revision, made almost all revisions from the article of the intellectual content. JG-D made a substantial contribution to the 2D and 3D design of the work, he made camera testing and part of data collection, he made the preliminary analysis, and earlier interpretation of data for the work. VF-B made time calibration of the overall experiment, made part of data collection, he helped with the preliminary analysis and he critically reviewed results and earlier discussion sections. LC-B arranged the physical initial test of the experiment, made part of data collection, she reviewed the initial draft and she critically reviewed the earlier results section. All authors contributed to the article and approved the submitted version.
Funding
The present research experiment was in part within the project “Testeo de Electrofisiología Cognitiva de atención humana adaptado a realidad aumentada” supported by the Scientific Research Institute of the Universidad de Lima. Also, the experiment of the present research and its interpretation was part of the current analysis of the project with the support of the Research Institute of the UNTELS for financing the project “Estudio de comportamiento ocular ante sismo: escenario simulado mediante realidad aumentada”, see RCO-230-2017-UNTELS (2017–2019). Research duties provided for the first author (CM-V) under “Decreto Supremo N.° 003-2018-MINEDU,” also for some time to amend interpretation at University of Delaware under the project “EEG and fMRI studies for dyslexia” in Center for Brain and Biomedical Imaging in Delaware in 2020, and UPSJB since 2022, for overall Markov brain proposal and writing-up.
Acknowledgments
We thank Teresa Quiroz and Juan Gutierrez for the encouragement of recordings at Universidad de Lima, as well as the continuous effort of Lupe Pizán (former Vice-Canchellor of UNTELS) to collaborate between institutions and to Carlos Vegas and Segundo León from UPSJB for the encouragement of expanding analysis to the human brain and improving discussion. Also to Professor Oscar Dall’Orto (RIP) for initial feedback on the experiment and to Andrés Mugruza for helping in the test of children participants to improve and simplify the experiment. Also to reviewers for their valuable concerns about reporting conclusions and improving discussions.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum.2022.955534/full#supplementary-material.
References
Alqithami, S. (2021). A serious-gamification blueprint towards a normalized attention. Brain Inform. 8:6. doi: 10.1186/s40708-021-00127-3
Alqithami, S., Alzahrani, M., Alzahrani, A., and Mostafa, A. (2019). “Modeling an augmented reality game environment to enhance behavior of ADHD patients,” in International Conference on Brain Informatics, eds P. Liang, V. Goel, and C. Shan (Switzerland: Springer Nature), 179–188. doi: 10.1007/978-3-030-37078-7_18
Baena-Mirabete, S., Espinal, A., and Puig, P. (2020). Exploring the randomness of mentally generated head-tail sequences. Stat. Model. 20, 225–248. doi: 10.1177/1471082X18816410
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Tb, D., et al. (2018). Distributed distributional deterministic policy gradients. arXiv [Preprint]. doi: 10.48550/arXiv.1804.08617
Batallones, A., Sanchez, K., Mott, B., Coffran, C., and Hsu, D. F. (2015). On the combination of two visual cognition systems using combinatorial fusion. Brain Inform. 2, 21–32. doi: 10.1007/s40708-015-0008-0
Busemeyer, J. R., and Townsend, J. T. (1992). Fundamental derivations from decision field theory. Math. Soc. Sci. 23, 255–282. doi: 10.1016/0165-4896(92)90043-5
Chen, H. Y., Dix, A., Goh, J. O. S., Smolka, M. N., Thurm, F., and Li, S. C. (2021). Effects and mechanisms of information saliency in enhancing value-based decision-making in younger and older adults. Neurobiol. Aging 99, 86–98. doi: 10.1016/j.neurobiolaging.2020.11.018
Chen, C. H., Ho, C.-H., and Lin, J. B. (2015). The development of an augmented reality game-based learning environment. Procedia Soc. Behav. Sci. 174, 216–220. doi: 10.1016/j.sbspro.2015.01.649
Cooney, S. M., Brady, N., and Ryan, K. (2017). Spatial orienting of attention to social cues is modulated by cue type and gender of viewer. Exp. Brain Res. 235, 1481–1490. doi: 10.1007/s00221-017-4909-4
Córdova Berríos, L. L. C., Mugruza-Vassallo, C. A., and Flores-Benites, V. (2018). “Missed signals in the congruency between visual distracting cues and auditory goals,” in Proceedings of the 8th International Conference on Information Communication and Management, (New York, NY: Association for Computing Machinery), 7–11. doi: 10.1145/3268891.3268904
Daglas, R., Allott, K., Yücel, M., Henry, L. P., Macneil, C. A., Hasty, M. K., et al. (2017). Cognitive functioning following stabilisation from first episode mania. Int. J. Bipolar Disord. 5:39. doi: 10.1186/s40345-017-0108-2
de Bruin, E. A., Bijl, S., Stam, C. J., Böcker, K. B., Kenemans, J. L., and Verbaten, M. N. (2004). Abnormal EEG synchronisation in heavily drinking students. Clin. Neurophysiol. 115, 2048–2055. doi: 10.1016/j.clinph.2004.04.010
Engbert, R., and Kliegl, R. (2003). Microsaccades uncover the orientation of covert attention. Vis. Res. 43, 1035–1045. doi: 10.1016/s0042-6989(03)00084-1
Fan, J., McCandliss, B. D., Fossella, J., Flombaum, J. I., and Posner, M. I. (2005). The activation of attentional networks. Neuroimage 26, 471–479. doi: 10.1016/j.neuroimage.2005.02.004
Forth, J., Agres, K., Purver, M., and Wiggins, G. A. (2016). Entraining IDyOT: timing in the information dynamics of thinking. Front. Psychol. 7:1575. doi: 10.3389/fpsyg.2016.01575
Hmamouche, Y., Ochs, M., Prevot, L., and Chaminade, T‥ (2020). “Exploring the dependencies between behavioral and neuro-physiological time-series extracted from conversations between humans and artificial agents,” in 9th International Conference on Pattern Recognition Applications and Methods, (Valletta, Malta: SCITEPRESS-Science and Technology Publications), 353–360. doi: 10.5220/0008989503530360
Ibrahim, E. N. M., Jamali, N., and Suhaimi, A. I. H. (2021). Exploring gamification design elements for mental health support. Int. J. Adv. Technoloration Eng. Exploration 8, 114–125. doi: 10.19101/IJATEE.2020.S1762123
Imbert, N., Vignat, F., Kaewrat, C., and Boonbrahm, P. (2013). Adding physical properties to 3D models in augmented reality for realistic interactions experiments. Procedia Comput. Sci. 25, 364–369. doi: 10.1016/j.procs.2013.11.044
Judelson, D. A., Armstrong, L. E., Sökmen, B., Roti, M. W., Casa, D. J., and Kellogg, M. D. (2005). Effect of chronic caffeine intake on choice reaction time, mood and visual vigilance. Physiol. Behav. 85, 629–634. doi: 10.1016/j.physbeh.2005.06.011
Khalvati, K., Park, S. A., Mirbagheri, S., Philippe, R., Sestito, M., Dreher, J.-C., et al. (2019). Modeling other minds: Bayesian inference explains human choices in group decision-making. Sci. Adv. 5:eaax8783. doi: 10.1126/sciadv.aax8783
Kodirov, E., Xiang, T., Fu, Z., and Gong, S. (2016). “Person re-identification by unsupervised l1 graph learning,” in European Conference on Computer Vision, (Cham: Springer), 178–195. doi: 10.1007/978-3-319-46448-0_11
Koechlin, E., Ody, C., and Kouneiher, F. (2003). The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185. doi: 10.1126/science.1088545
Leeb, R., Lancelle, M., Kaiser, V., Fellner, D. W., and Pfurtscheller, G. (2013). Thinking penguin: multimodal brain-computer interface control of a VR game. IEEE Trans. Comput. Intell. AI Games 5, 117–128. doi: 10.1109/TCIAIG.2013.2242072
Mary, A., Dayan, J., Leone, G., Postel, C., Fraisse, F., Malle, C., et al. (2020). Resilience after trauma: the role of memory suppression. Science 367:eaay8477. doi: 10.1126/science.aay8477
Mayas, J., Parmentier, F. B., Andres, P., and Ballesteros, S. (2014). Plasticity of attentional functions in older adults after non-action video game training: a randomized controlled trial. PLoS One, 9:e92269. doi: 10.1371/journal.pone.0092269
Menozzi, M., Bergande, E., Bonan, J., Sury, P., and Krueger, H. (2007). Minimum amount of time required to retrieve peripheral numeric information in augmented-reality systems using a colour sequential DLP display. Displays 28, 85–91. doi: 10.1016/j.displa.2007.04.005
Mitsuda, T., Otani, M., and Sugimoto, S. (2019). Gender and individual differences in cueing effects: visuospatial attention and object likability. Attent. Percept. Psychophys. 81, 1890–1900. doi: 10.3758/s13414-019-01743-2
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., et al. (2016). Asynchronous methods for deep reinforcement learning,” in International Conference on Machine Learning, (New York, NY: JMLR.org), 1928–1937.
Mugruza-Vassallo, C. A. (2016a). “Database methodology for therapy evaluation in auditory schizophrenia disorder based on continuity evolution of symptoms,” in 2016 6th International Conference on Information Communication and Management (ICICM) (Hatfield, UK: IEEE), 298–303. doi: 10.1109/INFOCOMAN.2016.7784261
Mugruza-Vassallo, C. A. (2016b). “Different regressors for linear modelling of ElectroEncephaloGraphic recordings in visual and auditory tasks,” in 2016 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN) (Hatfield, UK: IEEE), 260–265. doi: 10.1109/BSN.2016.7516270
Mugruza-Vassallo, C. A., and Potter, D. D. (2019). Context dependence signature, stimulus properties and stimulus probability as predictors of ERP amplitude variability. Front. Hum. Neurosci. 13:39. doi: 10.3389/fnhum.2019.00039
Mugruza-Vassallo, C. A., Potter, D. D., Tsiora, S., Macfarlane, J. A., and Maxwell, A. (2021). Prior context influences motor brain areas in an auditory oddball task and prefrontal cortex multitasking modelling. Brain Inform. 8:5. doi: 10.1186/s40708-021-00124-6
Mühlberg, S., and Müller, M. M. (2020). Alignment of continuous auditory and visual distractor stimuli is leading to an increased performance. Front. Psychol. 11:790. doi: 10.3389/fpsyg.2020.00790
Obeso, I., Herrero, M. T., Ligneul, R., Rothwell, J. C., and Jahanshahi, M. (2021). A causal role for the right dorsolateral prefrontal cortex in avoidance of risky choices and making advantageous selections. Neuroscience 458, 166–179. doi: 10.1016/j.neuroscience.2020.12.035
Parmentier, F. B., Elsley, J. V., and and Ljungberg, J. K. (2010). Behavioral distraction by auditory novelty is not only about novelty: the role of the distracter’s informational value. Cognition 115, 504–511. doi: 10.1016/j.cognition.2010.03.002
Posner, M. I. (1980). Orientation of attention. The VIIth sir frederic bartlett lecture. Q. J. Exp. Psychol. 32A, 3–25.
Posner, M. I., Walker, J. A., Friedrich, F. J., and Rafal, R. D. (1984). Effects of parietal injury on covert orienting of attention. J. Neurosci. 4, 1863–1874. doi: 10.1523/JNEUROSCI.04-07-01863.1984
Potter, D. D., Bassett, M. R., Jory, S. H., and Barrett, K. (2001). Changes in event-related potentials in a three-stimulus auditory oddball task after mild head injury. Neuropsychologia 39, 1464–1472. doi: 10.1016/s0028-3932(01)00057-4
Rosburg, T. (2018). Auditory N100 gating in patients with schizophrenia: a systematic meta-analysis. Clin. Neurophysiol. 129, 2099–2111. doi: 10.1016/j.clinph.2018.07.012
Rousselet, G. A., Pernet, C. R., Bennett, P. J., and Sekuler, A. B. (2008). Parametric study of EEG sensitivity to phase noise during face processing. BMC Neurosci. 9: 98. doi: 10.1186/1471-2202-9-98
Samrani, G., Marklund, P., Engström, L., Broman, D., and Persson, J. (2018). Behavioral facilitation and increased brain responses from a high interference working memory context. Sci. Rep. 8:15308. doi: 10.1038/s41598-018-33616-3
Savalia, T., Shukla, A., and Bapi, R. S. (2016). A unified theoretical framework for cognitive sequencing. Front. Psychol. 7:1821. doi: 10.3389/fpsyg.2016.01821
Schneider, F., Horowitz, A., Lesch, K. P., and Dandekar, T. (2020). Delaying memory decline: different options and emerging solutions. Transl. Psychiatry 10:13. doi: 10.1038/s41398-020-0697-x
Shahal, A. (2018). Adjusting displays on user monitors and guiding users’ attention. U.S. Patent Application No. 15/759,229.
Shi, W., Yang, F., Xia, J., and Wang, M. (2019). Differentiating the effects of anger and sadness: a perspective of spontaneous trait inference. Int. J. Psychol. 54, 487–494. doi: 10.1002/ijop.12491
Soon, C. S., He, A. H., Bode, S., and Haynes, J. D. (2013). Predicting free choices for abstract intentions. Proc. Natl. Acad. Sci. U S A 110, 6217–6222. doi: 10.1073/pnas.1212218110
Sun, F., Zhang, W., Chen, J., Wu, H., Tan, C., and Su, W. (2018). Fused fuzzy petri nets: a shared control method for brain-computer interface systems. IEEE Trans. Cogn. Dev. Sys. 11, 188–199. doi: 10.1109/TCDS.2018.2818173
Taraghi, B., Ebner, M., Saranti, A., and Schon, M. (2014). “On using markov chain to evidence the learning structures and difficulty levels of one digit multiplication” in Proceedings of the Fourth International Conference on Learning Analytics and Knowledge, (New York, NY: Association for Computing Machinery), 68–72. doi: 10.1145/2567574.2567614
Temple, J. L., Bernard, C., Lipshultz, S. E., Czachor, J. D., Westphal, J. A., and Mestre, M. A. (2017). The safety of ingested caffeine: a comprehensive review. Front. Psychiatry 8:80. doi: 10.3389/fpsyt.2017.00080
Tomov, M. S., Schulz, E., and Gershman, S. J. (2021). Multi-task reinforcement learning in humans. Nat. Hum. Behav. 5, 764–773. doi: 10.1038/s41562-020-01035-y
Torres-Tejada, S., Portilla-Fernández, J. A., Mugruza-Vassallo, C. A., and Córdoba-Berrios, L. L. (2020). Variations of reaction times explained by stimuli changes in size and perspective in 2D and 3D for selective attention. Revista mexicana de ingeniería biomédica 41, 91–104. doi: 10.17488/rmib.41.1.7
Vasilev, M. R., Parmentier, F. B., Angele, B., and Kirkby, J. A. (2019). Distraction by deviant sounds during reading: an eye-movement study. Q. J. Exp. Psychol. (Hove) 72, 1863–1875. doi: 10.1177/1747021818820816
Vuforia. (2012). Vuforia is the leading AR platform. Vuforia. Available online at: https://www.vuforia.com/ Accessed June 14, 2017.
Welford, A. T. (1988). Reaction time, speed of performance and age. Ann. N Y Acad. Sci. 515, 1–17. doi: 10.1111/j.1749-6632.1988.tb32958.x
Wu, L., Wang, Y., Gao, J., Wang, M., Zha, Z. J., and Tao, D. (2021). Deep coattention-based comparator for relative representation learning in person re-identification. IEEE Trans. Neural Netw. Learn. Syst. 32, 722–735. doi: 10.1109/TNNLS.2020.2979190
Wu, L., Wang, Y., Li, X., and Gao, J. (2018). What-and-where to match: deep spatially multiplicative integration networks for person re-identification. Pattern Recogn. 76, 727–738. doi: 10.1016/j.patcog.2017.10.004
Keywords: augmented reality, cognitive neuroscience, decision task, environment design, Go-Go task, Markov chain, neurocomputational models, video games
Citation: Mugruza-Vassallo CA, Granados-Domínguez JL, Flores-Benites V and Córdova-Berríos L (2022) Different Markov chains modulate visual stimuli processing in a Go-Go experiment in 2D, 3D, and augmented reality. Front. Hum. Neurosci. 16:955534. doi: 10.3389/fnhum.2022.955534
Received: 28 May 2022; Accepted: 11 October 2022;
Published: 21 November 2022.
Edited by:
Geng Chen, Northwestern Polytechnical University, ChinaReviewed by:
Liuting Diao, Ningbo University, ChinaXuyun Wen, Nanjing University of Aeronautics and Astronautics, China
Copyright © 2022 Mugruza-Vassallo, Granados-Domínguez, Flores-Benites and Córdova-Berríos. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Carlos Andrés Mugruza-Vassallo, Y211Z3J1emFAeWFob28uY29t