Examining virtual driving test performance and its relationship to individuals with HIV-associated neurocognitive disorders

Significance: Existing screening tools for HIV-associated neurocognitive disorders (HAND) are often clinically impractical for detecting milder forms of impairment. The formal diagnosis of HAND requires an assessment of both cognition and impairment in activities of daily living (ADL). To address the critical need for identifying patients who may have disability associated with HAND, we implemented a low-cost screening tool, the Virtual Driving Test (VDT) platform, in a vulnerable cohort of people with HIV (PWH). The VDT presents an opportunity to cost-effectively screen for milder forms of impairment while providing practical guidance for a cognitively demanding ADL.

Objectives: We aimed to: (1) evaluate whether VDT performance variables were associated with a HAND diagnosis and if so; (2) systematically identify a manageable subset of variables for use in a future screening model for HAND. As a secondary objective, we examined the relative associations of identified variables with impairment within the individual domains used to diagnose HAND.

Methods: In a cross-sectional design, 62 PWH were recruited from an established HIV cohort and completed a comprehensive neuropsychological assessment (CNPA), followed by a self-directed VDT. Dichotomized diagnoses of HAND-specific impairment and impairment within each of the seven CNPA domains were ascertained. A systematic variable selection process was used to reduce the large amount of VDT data generated, to a smaller subset of VDT variables, estimated to be associated with HAND. In addition, we examined associations between the identified variables and impairment within each of the CNPA domains.

Results: More than half of the participants (N = 35) had a confirmed presence of HAND. A subset of twenty VDT performance variables was isolated and then ranked by the strength of its estimated associations with HAND. In addition, several variables within the final subset had statistically significant associations with impairment in motor function, executive function, and attention and working memory, consistent with previous research.

Conclusion: We identified a subset of VDT performance variables that are associated with HAND and assess relevant functional abilities among individuals with HAND. Additional research is required to develop and validate a predictive HAND screening model incorporating this subset.

Introduction

In the era of antiretroviral therapy, the most severe form of HAND (HIV-associated dementia) is rare (∼2% prevalence). Although less severe, functional impairment associated with milder forms of HAND (∼15–60% prevalence in PWH) is more common (Saylor et al., 2016; World Health Organization, 2017; Yusuf et al., 2017). Failure to detect HAND early is associated with a lower quality of life and decreased survival (Heaton et al., 2004; Antinori et al., 2013; Vance et al., 2014; Kronemer et al., 2017). Earlier detection of HAND through implementation of screening tools in the outpatient clinic is a diagnostic goal worldwide; however, previously tested screening tools are limited in their sensitivity and specificity to detect these milder forms of HAND (Roebuck-Spencer et al., 2017).

In part, due to new reimbursable billing codes, administration of cognitive screening tests have emerged (Abraham, 2013; Federal Register., 2015; Roebuck-Spencer et al., 2017); however, significant limitations in these tools for HAND have been identified (Roebuck-Spencer et al., 2017). The existing tools (1) are not self-directed and require skilled staff for administration and interpretation (commonly not found outside of research protocols) (Antinori et al., 2013); (2) lack face validity to assess common functional activities of daily living (ADL) (Antinori et al., 2013); and (3) have failed to meet acceptability for sensitivity and specificity and have limited value for detecting milder forms of HAND [e.g., IHDS (Sacktor et al., 2005; Chan et al., 2019), MoCA (Rosca et al., 2019), MMSE (de Souza et al., 2016), SSQ (de Souza et al., 2016), and CAT-Rapid (Antinori et al., 2007, 2013)]. Additionally, the lack of ecological validity of current screening tools limits their value beyond screening: these tools provide limited insight on ADLs and therefore, cannot provide practical guidance on addressing potential impairment (Antinori et al., 2013). Thus, there is a need for screening tools that are rapid, cost-effective, and reliable for detecting milder forms of HAND, while providing practical guidance on ADL.

Driving is cognitively demanding (Karthaus and Falkenstein, 2016) and an instrumental ADL (Green Car Reports, 2014; Brenan, 2018), which we hypothesize, can serve as a platform for the earlier detection of milder forms of HAND. Safe driving requires efficient multisensory processing and integration, motor control of the vehicle, sustained attention, and related cognitive abilities to manage distractions, maintain good situational awareness, and to make quick decisions and rapid responses in critical moments to avoid collisions (Wadley et al., 2009; Walshe et al., 2017). Thus, driving requires many of the cognitive abilities typically affected by HAND, such as complex attention, executive function, and fine motor speed and dexterity (Roenker et al., 2003; Wadley et al., 2009; Karthaus and Falkenstein, 2016; Walshe et al., 2017). Previous studies have demonstrated impaired driving performance among those with HAND: PWH with cognitive impairments had slower reaction times, as compared to normative controls (Vance et al., 2014), increased time requirements for completion of the driving simulation task (Foley et al., 2013), impaired route planning (Foley et al., 2013), and an increased risk for traffic collisions (Marcotte et al., 1999, 2004; Gouse et al., 2021). However, previous applications of driving assessments in PWH were not scalable for routine clinical settings, as they utilized on-road evaluations or laboratory-grade equipment.

In a novel application, this research program aims to utilize a well-established virtual driving test (VDT) platform (Walshe et al., 2020) that has been validated to predict on-road performance (Winston et al., 2019; Grethlein et al., 2020) as a probe to screen for neurocognitive impairment among PWH. The currently used HAND screening tools (e.g., IHDS, MoCA, MMSE, SSQ, CAT-Rapid) may have minimal direct costs associated with them (e.g., cost to purchase license for use, cost to purchase equipment, if any); however, there is an indirect cost of administering these tests in routine clinical practice (e.g., trained staff labor). We do not argue that the VDT would be any less expensive than existing tools; however, our experience of using the VDT as a platform to assess safe driving skills in the commercial market demonstrates that our delivery costs for the VDT is less than $2.00/test (inclusive of hardware, administration, and software license) when used at scale.

Given that the VDT platform is commercially used as a rapid and low-cost assessment to test complicated tasks associated with safe driving, it has the potential to also be used to screen for HAND (using driving as a probe). As a first step in evaluating the potential for the VDT to be used as a HAND screening tool, this preliminary study aimed to (1) evaluate whether VDT performance variables were associated with a HAND diagnosis and if so; (2) systematically identify a manageable subset of variables for use in a future screening model for HAND. As a secondary objective, we examined the relative associations of identified variables with impairment within seven individual domains used to diagnose HAND, to better understand the impact of impairment on a highly common ADL.

Materials and methods

Study participants and eligibility criteria

In a cross-sectional design, participants were recruited from the Drexel University Comprehensive NeuroAIDS Center (CNAC) cohort (supported by NIMH P30MH092177) over an 8-month period (November 2020–May 2021). Following a chart review, PWH on antiretroviral therapy, with a previous comprehensive neuropsychological assessment (CNPA), and previous CNAC research experience, were invited to participate. Study participants were recruited and screened by a telephone interview and provided verbal consent. Table 1 depicts the study’s exclusion criteria, which were selected to minimize confounding factors and isolate HAND-specific neuropsychological impairment (Tedaldi et al., 2015). Having a valid driver’s license was not used as an exclusion criterion, given the sociodemographic characteristics of the CNAC cohort (urban, low income, where suspended or expired driver’s licenses are more common, but driving is not a foreign activity). All study procedures were approved by the Drexel University Institutional Review Board and were completed on Drexel University School of Medicine premises (Philadelphia, Pennsylvania).

TABLE 1

Table 1. Study exclusion criteria.

Study visit procedures

Clinical eligibility criteria were ascertained prior to the telephone interview via a medical chart review. The telephone interview was used to confirm eligibility criteria (except for the urine drug screen), obtain verbal consent, and schedule the study visit. During the study visit, written informed consent was obtained and participants were then asked to complete a urine drug screen examination. Participants that tested positive for cocaine or illegal opioid use were excluded from subsequent study visit procedures. Marijuana (medical or recreational) use was not an exclusion criterion as its use is very prevalent in the community. Additionally, marijuana use can yield a positive urine test for up to 30 days, whereas a positive result for other drugs of abuse typically indicates more recent use (e.g., in the past 3 days for cocaine). All other participants proceeded to complete a self-report survey (to collect driving history and other demographic information), a CNPA, and finally, the VDT.

Administering the comprehensive neuropsychological assessment

The CNPA (lasting approximately 1.5–2 h) was administered by a trained psychometrist (blinded to all other study data collected) to confirm the presence or absence of HAND through an assessment of seven domains, using at least two measures per domain (Table 2) in accordance with Frascati criteria (Antinori et al., 2007). Tests were selected for their sensitivity to HAND (Carey et al., 2004a,b; Woods et al., 2004; Monteiro de Almeida et al., 2016).

TABLE 2

Table 2. Component tests of the CNPA.

Administering the virtual driving test

The VDT (Ready-Assess™; ready-assess.com) is a well-established driving assessment platform used in both commercial and clinical settings (Walshe et al., 2020; Lee, 2021). In a self-directed workflow (containing both voiceover prompts and on-screen navigational instructions), study participants were exposed to ecologically valid crash scenarios (McDonald et al., 2014) contained in a typical driving route, lasting approximately 15 min. Upon completing a brief eye-tracking calibration step (Tobii 5¹), study participants were then asked to independently complete the VDT by logging into the VDT workstation with their assigned research identification number. Designed to be installed in a limited-space environment (e.g., a small office desk), the VDT workstation included an internet-connected Windows PC, a Logitech G29 (steering wheel and pedals) and a pair of over-the-ear headphones. Upon logging into the VDT software installed on the workstation, each participant was exposed to three primary modules in a self-directed manner. In the first module, an animated orientation video was presented, lasting approximately 30 s. The purpose of this video was to provide the participant a basic introduction to the VDT. The second module incorporated an introductory drive. During this module, the participant was introduced to the different VDT control inputs (e.g., steering, brake, throttle, turn signal, transmission, etc.). Additionally, the participant was asked to complete a series of basic driving maneuvers (accelerating, braking, steering through a curve and executing a 90-degree turn) to acclimate themselves to the program. This module lasted approximately 3 min and allowed the participant to repeat the process if needed. The final module was a virtual driving test route which was scored for performance. The route contained variations of common crash scenarios as described in McDonald et al. (2014) and common driving tasks and roadway hazards (e.g., pedestrian crosswalks, school zones, ambulances, etc.), embedded in a typical driving route, lasting approximately 10 min. Each participant was exposed to the same VDT driving route (same tasks and exposures in the same order).

Data preparation

Confirming the presence of HIV-associated neurocognitive disorders

Raw, unadjusted scores from the individual CNPA tests were converted to demographically adjusted (age, gender, education, and race) T-scores using published norms based on HIV-seronegative healthy populations (Strauss et al., 2006; Norman et al., 2011; Schretlen et al., 2018). T-scores for each test were then converted to deficit scores, and a deficit score for each domain (average of deficit scores for all tests in the domain) and a Global Deficit Score (GDS: average of domain deficit scores) were computed for each participant using established methods (Carey et al., 2004a; Blackstone, 2012). A GDS cutoff of ≥ 0.5 was used to indicate the presence of HAND (Carey et al., 2004a). In addition to HAND status, a cutoff of ≥ 1 was applied to domain deficit scores to indicate the presence of impairment in that domain. Standard three-level GDS-defined HAND categories (Antinori et al., 2007) were also computed, however, were not used in the primary analysis due to HIV-associated dementia being relatively rare in this cohort. The GDS approach to HAND classification was utilized because recent evidence demonstrates that GDS-based HAND diagnoses are more specific than Frascati criteria-based diagnoses involving the assessment of functional impairments of ADL (Heaton, 2021).

Virtual driving test data processing

Each VDT session generated a JSON file (referred to as a replay file) containing a 10 Hz multivariate, time series record. The replay file was used to compute 40 unique time series channels to measure driving performance across multiple driving elements (e.g., speed, lane deviation, following distance, elapsed time). Within the 40 time series channels, 8 eye-tracking channels were computed (using an automated, time-synchronized integration procedure developed by Diagnostic Driving, Inc.) to quantify eye-gaze movement during the VDT session. These channels were ultimately converted into unique variables (e.g., min, max, mean, median, standard deviation of each channel) and further organized into global (across the entire VDT route) and zone-specific variables (within 22 unique segments of the driving route). In total, 2,921 time-series-derived variables were computed for each study participant’s VDT session (previously described in Grethlein et al., 2020). In addition to the variables described above, 8 higher-order errors (e.g., counts of collisions, red light errors, stop sign errors, navigational errors, etc.) were automatically recorded in the replay file to compute a composite score of VDT performance (VDT Error Score) based on a linear combination of these higher-order error counts. This composite score of performance has been previously used to predict on-road examination results for licensure (Winston et al., 2019). In total, a completed VDT session yielded 2,601 explanatory variables, including VDT Error Score (after removing variables with zero variance). The relatively large number of variables with zero variance was expected due to the number of unique, evaluable zones along the VDT driving route (e.g., we would expect zero pedestrian collisions in a zone without pedestrians). Additionally, it is acknowledged that several multi-collinear relationships exist among these variables. No additional data processing was conducted (scaling of variables or removal of outlier cases) prior to analysis.

Final analytical dataset and analytical methods

A final analytical dataset consisting of 2,601 explanatory variables and one binary outcome variable for HAND status was generated, yielding a matrix of size nx2602, where n would be the final sample size. In addition, seven binary variables were generated to indicate impairment within each of the seven CNPA domains.

Summary of supervised variable selection methods used

Given the small sample size and high dimensionality of the final dataset, systematic reduction methods were required to identify which VDT variables, if any, were associated with HAND. Our strategy was to utilize a methodology that would allow for maximum interpretability of results (as the goal of the VDT is designed to provide practical guidance on safe driving), thereby ruling out common reduction techniques such as principal component analysis that transform input variables into a potentially less intuitive space by aggregation of variables.

In order to overcome the multiple comparisons problem for this small dataset, our analysis leverages re-samplings of the dataset similar to stratified fivefold cross-validation: splitting the labeled samples into 5 disjoint folds, ideally preserving the ratio of samples with and without HAND in each split. Splitting the dataset combined with non-parametric Kruskal-Wallis (KW) tests, was used to provide five opportunities to independently test each of the 2,601 individual hypotheses and then rank each variable during a two-stage variable selection process (using the KW H-value, see Figure 1). Specifically, we generated five different training sets (but we did not make use of the five test sets) to alleviate concerns around identifying false-positive variables (a limitation of multiple hypothesis testing).

FIGURE 1

Figure 1. Analytical workflow for dimensionality reduction via a two-stage variable selection.

With these five training sets, we obtained five independently generated lists of selected VDT performance variables; cross-examining which variables were found to be consistently associated with HAND in all five splits to build a consensus. A random seed set to 0 was used for the stratified splitting procedure (for reproducibility). Additionally, the stability and robustness of variable selection results were evaluated using “top k-Lists” normalized Kendall rank correlation (p = 0.5 “neutral approach”; Fagin et al., 2003) and Jaccard similarity (Saeys et al., 2008), to demonstrate that variables were not selected due to random chance or overfitting to a sub-sample of participants produced from splitting. All analyses were conducted using Python (version 3.8.9)² and R (version 4.0.2).³

Variable selection and forming a final consensus

Using the training data (a unique subset of fourfold of VDT data) in each split, we computed the univariate KW H-value for each of the 2,601 VDT variables relative to HAND status. The VDT variables in each split were then ranked in descending order of univariate KW H-value. In all experiments conducted, the “top ranked variable” would have a rank of 1 (inferring the VDT variable most associated with HAND status), and the lowest ranked VDT variable would have a rank of 2601. Ranking ties were handled by deferring to the VDT variable that appeared first in the original dataset (from the start of the drive). In each split we extracted the “top k variables” ranked list of selected VDT variables.

By experimental design, the “top k variables” selected in one split were not dependent on the “top k variables” selected from any other split, though the training data from any two splits shared roughly 50% of the same VDTs (twofold of overlap, similar to fivefold cross-validation). A second-stage filtering criterion was imposed so that only VDT variables that were consistently an element of the “top k variables” ranked lists in all 5 splits would be included in the consensus, i.e., the consensus was formally defined as the set intersection of the 5 “top k variables” ranked lists. For each VDT variable in the consensus, we computed the median KW H-value across the 5 splits, estimating how associated an individual VDT variable was in at least three of the five splits. These median KW H-values were used to rank the consensus in descending order. There was no guarantee that such an intersection would be non-empty, as the likelihood of any single VDT variable being consistently ranked in the “top k variables” in all 5 splits drops as k decreases.

To choose k prior to producing the final rankings, we tested integer values of k ∈ [1, 1,000] as thresholds for forming consensus using the KW H-value as the ranking score. By monitoring the mean KW H-Value of the “top k variables” selected in each split, we evaluated the relative trade-off between selecting a smaller number of supposedly informative VDT variables and balancing the diminishing returns of including a greater number of less informative variables into the selected group. We used the “kneed” python package (Satopa et al., 2011) to find the “elbow point” value k for each split to estimate the exact value of k where the trade-off stops working in our favor. In most cases, the “elbow point” for the value of k was between 150 and 200 (results omitted for brevity). Using the “elbow point” results to guide our choice of k, we chose to conservatively build our consensus using the k = 100 top ranked VDT variables from all five splits for the HAND outcome variable. This was done to cast a wide enough net to allow for some VDT variables to make it into the consensus, while using intermediary ranking lists of constant length that are not so large as to allow uninformative VDT variables into the consensus.

Final variable categorizations

We utilized the authorship team’s subject matter expertise to review the final ranked-consensus list of VDT variables and assign each variable to a measured driving domain(s). Furthermore, and as part of our secondary aim, each variable in the final subset was evaluated to measure its relative association with impairment in each of the seven cognitive domains (independently assessed during the CNPA; Table 2) using a Spearman correlation matrix with a Bonferroni correction.

Results

Final analytical sample and characteristics

Seventy-nine (79) participants scheduled a study visit after providing verbal consent. Among these, 9 did not show up for their study visit and 1 participant was further excluded from subsequent procedures after testing positive during their urine drug screen examination. In addition, 1 participant declined to participate in subsequent study procedures after their negative urine drug screen (did not complete CNPA or VDT), 1 participant had an invalid CNPA, 1 participant could not complete the VDT due to feeling dizzy, and 4 participants could not complete the VDT due to network or equipment issues. A final study sample of sixty-two (62) participants met all study eligibility criteria and completed all study visit procedures. Among the final 62 participants, 37% (n = 23) tested positive for allowable drug use (prescribed pain management, psychiatric medication, or marijuana—common in this population). Of note, compared to participants who tested negative for allowable drug use, participants who tested positive, did not perform differently in the VDT (as measured by VDT Error Score; W = 404; p = 0.52) and did not have a different prevalence of HAND (χ² = 7.3*10^–5; p = 0.99). The demographic and clinical characteristics of the final study sample are detailed in Table 3 and generally matched the overall characteristics of the entire CNAC cohort.

TABLE 3

Table 3. Demographic and clinical characteristics of the final study sample.

More than half of the final study sample (N = 35; 57%) had a confirmed presence of HAND, as determined by the CNPA. Among those with HAND, 83% (n = 29) had either asymptomatic or mild impairment, and 17% (n = 6) had severe impairment (HIV-associated dementia). The prevalence of impairment within the seven cognitive domains assessed during the CNPA is detailed in Table 4. There were no statistically significant differences between having a HAND diagnosis and the demographic and clinical characteristics of the study participants. There were statistically significant differences between VDT Error Score and these characteristics. However, not having a current and valid driver’s license was significantly associated with having HAND [Note: Pennsylvania is a mandatory medical reporting state for driving privileges (Pennsylvania Department of Transportation, 2022)].

TABLE 4

Table 4. Prevalence of within-domain impairment among the study participants.

Variable selection results

Using the fivefold consensus approach, we were able to identify a final subset of 20 VDT variables that were associated with HAND status (Table 5). The consensus identified a mixture of six global (across the entire VDT) variables and fourteen zonal (within specific segments of the VDT route) variables. Of particular interest, the composite score of cumulative serious errors, VDT Error Score, was identified in the final subset (median H-value of 7.33, fivefold median p = 0.007) and was statistically associated with impairment of motor function, executive function, attention and working memory, and verbal memory. VDT performance variables that were statistically associated with a cognitive domain before or after the Bonferonni correction are reported in Table 5.

TABLE 5

Table 5. The final 20 ranked consensus variables associated with HAND status.

The distributions of each of the 20 identified VDT variables was examined and described in Table 6 to identify practical differences between participants with and without HAND.

TABLE 6

Table 6. Rank-ordered distributions of the 20 identified VDT variables.

Stability of HIV-associated neurocognitive disorders rankings

On average, any generated pair of “top 100 variables” rankings lists shared 42 VDT variables in common (average Jaccard similarity of 0.27) and had a normalized Kendall correlation value of less than 0.15. Both quantities indicated that similarly ranked subsets of VDT variables were consistently selected as part of the “top 100 variables” intermediate rankings. If the probability for any single VDT variable being selected in the “top 100 variables” rankings list was uniformly distributed across all 2,601 variables, then any given variable had a probability of 8.40*10^–8 of being selected independently in five such ranked lists. Figure 2 depicts comparisons of the “top 100 variables” rankings from all five splits utilized.

FIGURE 2

Figure 2. The Jaccard similarity score comparing all pairs of “top 100 variables” rankings lists, with the true size of the intersection (number of shared VDT performance variables selected in both intermediate rankings lists) annotated in each cell.

Discussion

This study presents an initial examination of VDT performance among PWH and its relationship with HAND and impairment within seven domains assessed during the CNPA. Applying a non-traditional and systematic data reduction method to a novel application, we were able to identify a smaller and stable subset of VDT performance variables, derived from a consensus generation process. Reducing the dimensionality of the dataset (via variable selection), allowed us to parsimoniously measure VDT performance and its relationship with HAND. Additionally, stability results provide further support that the variables were not selected due to random chance or overfitting to a sub-sample of participants. Rather, the stability results indicate that the consensus represents a reliable subset of VDT variables that could aid in the detection of individuals with HAND. Using this approach, 20 VDT variables were identified in the subset, representing a range of variables across the entire VDT route (global) and within specific regions and tasks along the route (zonal). The identification of several variables measuring vehicle control, road positioning, attention maintenance, and hazard anticipation and response corroborate previous findings examining deteriorated driving performance among individuals with HAND (Marcotte et al., 1999, 2004, 2006; Foley et al., 2013; Vance et al., 2014; Gouse et al., 2021). Furthermore, the inclusion of the composite VDT Error Score in the final consensus provides initial evidence that the VDT can assess complex, cognitive functioning related to HAND. Participants with HAND had nearly twice the VDT Error Score than those without HAND, suggesting that the level of complex cognitive and sensorimotor control required to drive safely and avoid collisions was manifested in VDT performance. Individual zones such as the “playground curve” (left-leaning curved roadway with an active school playground on right side) and “railroad zone” (congested roadway with an inactive railroad crossing) were designed to simultaneously test a variety of safe driving domains (e.g., vehicle control, road positioning, hazard anticipation, and response). Results in Table 5 demonstrate that these complicated “zonal” scenarios within the VDT can also tax the complex cognitive control implicated in HAND, as the level of cognitive and sensorimotor control required to safely traverse these zones is diminished in participants with HAND (greater lane deviation, heading misalignment and distance from roadway center events). In this study, as an initial step toward developing a valid screen for HAND, we identified a group of VDT variables (and their underlying tasks) within the domains of motor function, executive function, and attention and working memory that support the use of the VDT platform for identifying clinically significant functional impairment associated with HAND. Our findings provide preliminary evidence that the VDT can probe complex cognitive and sensorimotor control involved with HAND and potentially serve as a low-cost screening tool to detect HAND earlier in the disease course. However, additional research is required to develop and validate a predictive HAND screening model incorporating the subset of VDT variables identified from this analysis.

Limitations

Our results are not without limitations. We did not include a healthy control group in the analysis as a mechanism to compare VDT performance among PWH who do not have HAND with a healthy population (HIV-negative and no neurocognitive impairment). Second, the ratio of sample size to evaluable variables was low. Our evaluation of “top 100 variables” ranking stability indicated that even within a small number of participants, there was significant variance in VDT performance in regards to HAND status. Third, we did not utilize “traditional” data reduction methods; rather, we designed our analytical methods to produce interpretable results (rankings of the VDT variables relative to HAND status). Fourth, our variable selection method did not identify any visual scanning variables (important in the evaluation of safe driving performance). This could be due to our limited sample size and our strict criteria for variable selection. Visual scanning metrics did appear in a final consensus of variables when restrictions on variable selection were eased. Fifth, the order of administering the CNPA and the VDT during the study visit was not randomized. All participants completed the CNPA before the VDT. This was due to clinical workflow constraints and the availability of the psychometrist. Sixth, 22% (n = 17) of the 79 participants who scheduled a study visit, did not complete all study procedures. Of these, the majority (n = 10; 59%) did not show for their study visit or declined study visit procedures. This is a common logistical limitation of conducting clinical research in this population. Finally, this study excluded individuals who tested positive for illegal substance abuse in order to better isolate HAND-specific impairment and test our primary hypotheses. Illegal substance abuse may be common among PWH and impact the broader applicability of using the VDT as a screen for HAND in this population. Future studies will incorporate participants who use illegal substances in order to better understand how it impacts VDT performance and its relationship with HAND.

Conclusion

This study identified a subset of VDT performance variables that are associated with HAND. The current findings have important implications for developing a new screening paradigm for HAND: using the VDT platform as a clinically feasible and ecologically valid assessment of function. However, additional research is needed to develop and validate a HAND screening model incorporating the final subset of VDT variables identified.

Data availability statement

The dataset presented in this article is not readily available because this research was funded by a Small Business Innovation Research grant (SBIR) from the National Institutes of Health (NIH) and awarded to Diagnostic Driving, Inc., with the specific purpose of commercializing intellectual property generated from this research program. Requests to access the dataset should be directed in writing to Diagnostic Driving, Inc. by DG, david@diagnosticdriving.com.

Ethics statement

The studies involving human participants were reviewed and approved by the Drexel University Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.

Author contributions

DG: responsible for VDT performance variables ranking analysis and overall manuscript preparation. VP: critical review and data collection methods. KD: CNPA methods and critical review. WD: CNPA methods and critical review of analytical methods. ZS, EW, DK, BA, KM, ST, BW, and FW: critical review. SO: critical review of analytical methods. VK: critical review and overall manuscript preparation. All authors contributed to the article and approved the submitted version.

Funding

This research was supported by federal research grants awarded by the National Institute of Mental Health: R43MH122035 (Principal Investigator: VK) and P30MH092177 (Principal Investigator: BW).

Acknowledgments

We would also like to acknowledge Luke Walker and Camden Segal for their hard work building and maintaining the VDT Ready-Assess™ system.

Conflict of interest

DG was an employee and a vesting shareholder of Diagnostic Driving, Inc. (DDI) (creator of the virtual driving test used in this study). VK was an intellectual property and financial interest in DDI and serves as its President and Chief Executive Officer. DG and VK’s potential of conflict of interest is managed by a conflict management plan from DDI whereby DG and VK had no interactions with study participants (all field data collection procedures were carried out by Drexel University CNAC staff members on Drexel University premises) and all methods and analyses were reviewed and approved by outside consultants with no intellectual or financial interest in DDI (Nicolas Skuli, PhD; a Senior Research Investigator and Director of the Stem Cell and Xenograft Core at the University of Pennsylvania and SO, PhD; an Associate Professor at Drexel University). Additionally, FW and the Children’s Hospital of Philadelphia (CHOP) have an intellectual property and financial interest in DDI. FW served as DDI’s Chief Scientific Advisor. This potential conflict of interest is managed under a conflict-of-interest management plan from CHOP and the University of Pennsylvania whereby FW had no interaction with study participants or involvement in data collection procedures and all analyses were reviewed and approved by outside consultants with no intellectual or financial interests in DDI (John Bolte, a traffic injury researcher at the Ohio State University; and Nancy Kasam-Adams, a behavioral researcher at CHOP and the University of Pennsylvania).

The reviewer LF declared a shared affiliation with the authors FW and DK to the handling editor at the time of review.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^ gaming.tobii.com
2. ^ python.org
3. ^ r-project.org

References

Abraham, M. (2013). CPT 2014. Current procedural terminology. Chicago: American Medical Association.

Google Scholar

Antinori, A., Arendt, G., Becker, J. T., Brew, B. J., Byrd, D. A., Cherner, M., et al. (2007). Updated research nosology for HIV-associated neurocognitive disorders. Neurology 69, 1789–1799. doi: 10.1212/01.WNL.0000287431.88658.8b

CrossRef Full Text | Google Scholar

Antinori, A., Arendt, G., Grant, I., Letendre, S., Chair, Muñoz-Moreno, J. A., et al. (2013). Assessment, diagnosis, and treatment of HIV-associated neurocognitive disorder: A Consensus report of the mind exchange program. Clin. Infect. Dis. 56, 1004–1017. doi: 10.1093/cid/cis975

PubMed Abstract | CrossRef Full Text | Google Scholar

Blackstone, K. (2012). Defining neurocognitive impairment in HIV: Deficit scores versus clinical ratings. Clin. Neuropsychol. 26, 894–908. doi: 10.1080/13854046.2012.694479

PubMed Abstract | CrossRef Full Text | Google Scholar

Brenan, M. (2018). 83% of U.S. adults drive frequently; Fewer enjoy it a lot. Gallup. Available online at: https://news.gallup.com/poll/236813/adults-drive-frequently-fewer-enjoy-lot.aspx (accessed March 15, 2022).

Google Scholar

Carey, C. L., Woods, S. P., Gonzalez, R., Conover, E., Marcotte, T. D., Grant, I., et al. (2004a). Predictive validity of global deficit scores in detecting neuropsychological impairment in HIV infection. J. Clin. Exp. Neuropsychol. 26, 307–319. doi: 10.1080/13803390490510031

PubMed Abstract | CrossRef Full Text | Google Scholar

Carey, C. L., Woods, S. P., Rippeth, J. D., Gonzalez, R., Moore, D. J., Marcotte, T. D., et al. (2004b). Initial validation of a screening battery for the detection of HIV-associated cognitive impairment. Clin. Neuropsychol. 18, 234–248. doi: 10.1080/13854040490501448

CrossRef Full Text | Google Scholar

Chan, L. G., Ho, M. J., Lin, Y. C., Ong, Y., and Wong, C. S. (2019). Development of a neurocognitive test battery for HIV-associated neurocognitive disorder (HAND) screening: Suggested solutions for resource-limited clinical settings. AIDS Res. Ther. 16:9. doi: 10.1186/s12981-019-0224-4

PubMed Abstract | CrossRef Full Text | Google Scholar

de Souza, E. M., Buoniconti, C. S., Valim, F. C., and Moura, A. S. (2016). Risk factors for neurocognitive impairment in HIV-infected patients and comparison of different screening tools. Dement. Neuropsychol. 10, 42–46. doi: 10.1590/s1980-57642016dn10100008

PubMed Abstract | CrossRef Full Text | Google Scholar

Fagin, R., Kumar, R., and Sivakumar, D. (2003). Comparing top k lists. SIAM J. Discre. Math. 17, 134–160. doi: 10.1137/S0895480102412856

CrossRef Full Text | Google Scholar

Federal Register. (2015). Notice of proposed rulemaking, medicare physician fee schedule. Potentially misvalued codes identified by screen for high expenditures. Fed. Regist. 79, 67576–67577.

Google Scholar

Foley, J. M., Gooding, A. L., Thames, A. D., Ettenhofer, M. L., Kim, M. S., Castellon, S. A., et al. (2013). Visuospatial and attentional abilities predict driving simulator performance among older HIV-infected adults. Am. J. Alzheimers Dis. Other Demen. 28, 185–194. doi: 10.1177/1533317512473192

PubMed Abstract | CrossRef Full Text | Google Scholar

Gouse, H., Masson, C. J., Henry, M., Thomas, K. G. F., Robbins, R. N., Kew, G., et al. (2021). The impact of HIV-associated neurocognitive impairment on driving performance in commercial truck drivers. AIDS Behav. 25, 689–698. doi: 10.1007/s10461-020-03033-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Green Car Reports (2014). 1.2 billion vehicles on world’s roads now, 2 billion by 2035: Report. Available online at: https://www.greencarreports.com/news/1093560_1-2-billion-vehicles-on-worlds-roads-now-2-billion-by-2035-report (accessed March 15, 2022).

Google Scholar

Grethlein, D., Winston, F. K., Walshe, E., Tanner, S., Kandadai, V., and Ontañón, S. (2020). Simulator pre-screening of underprepared drivers prior to licensing on-road examination: Clustering of virtual driving test time series data. J. Med. Internet Res. 22:e13995. doi: 10.2196/13995

PubMed Abstract | CrossRef Full Text | Google Scholar

Heaton, R. K. (2021). CNS complications in PWH: Frascati approach revisited. Paper presented at the 1st biotypes of CNS complications in people living with HIV meeting, Philadelphia, PA.

Google Scholar

Heaton, R. K., Marcotte, T. D., Mindt, M. R., Sadek, J., Moore, D. J., Bentley, H., et al. (2004). The impact of HIV-associated neuropsychological impairment on everyday functioning. J. Int. Neuropsychol. Soc. 10, 317–331. doi: 10.1017/S1355617704102130

PubMed Abstract | CrossRef Full Text | Google Scholar

Karthaus, M., and Falkenstein, M. (2016). Functional changes and driving performance in older drivers: Assessment and interventions. Geriatrics 1:12. doi: 10.3390/geriatrics1020012

CrossRef Full Text | Google Scholar

Kronemer, S. I., Mandel, J. A., Sacktor, N. C., and Marvel, C. L. (2017). Impairments of motor function while multitasking in HIV. Front. Hum. Neurosci. 11:212. doi: 10.3389/fnhum.2017.00212

CrossRef Full Text | Google Scholar

Lee, J. (2021). Children’s hospital of philadelphia launches innovative virtual teen driving assessment program with $4 million grant from NJM insurance group. PRNewswire. Available online at: https://www.prnewswire.com/news-releases/childrens-hospital-of-philadelphia-launches-innovative-virtual-teen-driving-assessment-program-with-4-million-grant-from-njm-insurance-group-301308005.html (accessed March 15, 2022).

Google Scholar

Marcotte, T. D., Heaton, R. K., Wolfson, T., Taylor, M. J., Alhassoon, O., Arfaa, K., et al. (1999). The impact of HIV-related neuropsychological dysfunction on driving behavior. The HNRC group. J. Int. Neuropsychol. Soc. 5, 579–592. doi: 10.1017/S1355617799577011

PubMed Abstract | CrossRef Full Text | Google Scholar

Marcotte, T. D., Lazzaretto, D., Scott, J. C., Roberts, E., Woods, S. P., Letendre, S., et al. (2006). Visual attention deficits are associated with driving accidents in cognitively-impaired HIV-infected individuals. J. Clin. Exp. Neuropsychol. 28, 13–28. doi: 10.1080/13803390490918048

PubMed Abstract | CrossRef Full Text | Google Scholar

Marcotte, T. D., Wolfson, T., Rosenthal, T. J., Heaton, R. K., Gonzalez, R., and Ellis, R. J. (2004). A multimodal assessment of driving performance in HIV infection. Neurology 63, 1417–1422. doi: 10.1212/01.WNL.0000141920.33580.5D

PubMed Abstract | CrossRef Full Text | Google Scholar

McDonald, C. C., Curry, A. E., Kandadai, V., Sommers, M. S., and Winston, F. K. (2014). Comparison of teen and adult driver crash scenarios in a nationally representative sample of serious crashes. Accid. Anal. Prev. 72, 302–308. doi: 10.1016/j.aap.2014.07.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Monteiro de Almeida, S., Kamat, R., Cherner, M., Umlauf, A., Ribeiro, C. E., de Paula, P. A., et al. (2016). Improving detection of HIV-associated cognitive impairment: Comparison of the international HIV dementia scale and a brief screening battery. J. Acquir. Immune Defic. Syndr. 74:332. doi: 10.1097/QAI.0000000000001224

CrossRef Full Text | Google Scholar

Norman, M. A., Moore, D. J., Taylor, M., Franklin, D. Jr., Cysique, L., Ake, C., et al. (2011). Demographically corrected norms for African Americans and caucasians on the hopkins verbal learning test-revised, brief visuospatial memory test-revised, stroop color and word test, and the wisconsin card sorting test 64-card version. J. Clin. Exp. Neuropsychol. 33, 793–804. doi: 10.1080/13803395.2011.559157

PubMed Abstract | CrossRef Full Text | Google Scholar

Pennsylvania Department of Transportation (2022). Mandatory reporting. PennDOT Driver and Vehicle Services. Available online at: https://www.dmv.pa.gov/Information-Centers/Medical-Reporting/Pages/Mandatory-Reporting.aspx (accessed March 15, 2022).

Google Scholar

Roebuck-Spencer, T. M., Glen, T., Puente, A. E., Denney, R. L., Ruff, R. M., Hostetter, G., et al. (2017). Cognitive screening tests versus comprehensive neuropsychological test batteries: A national academy of neuropsychology education paper. Arch. Clin. Neuropsychol. 32, 491–498. doi: 10.1093/arclin/acx021

PubMed Abstract | CrossRef Full Text | Google Scholar

Roenker, D. L., Cissell, G. M., Ball, K. K., Wadley, V. G., and Edwards, J. D. (2003). Speed-of-processing and driving simulator training result in improved driving performance. Hum. Factors 45, 218–233. doi: 10.1518/hfes.45.2.218.27241

CrossRef Full Text | Google Scholar

Rosca, E. C., Albarqouni, L., and Simu, M. (2019). Montreal cognitive assessment (MoCA) for HIV-associated neurocognitive disorders. Neuropsychol. Rev. 29, 313–327. doi: 10.1007/s11065-019-09412-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Sacktor, N. C., Wong, M., Nakasujja, N., Skolasky, R. L., Selnes, O. A., Musisi, S., et al. (2005). The international HIV dementia scale: A new rapid screening test for HIV dementia. AIDS 19, 1367–1374.

PubMed Abstract | Google Scholar

Saeys, Y., Abeel, T., and Van de Peer, Y. (2008). “Robust feature selection using ensemble feature selection techniques,” in Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, eds W. Daelemans, B. Goethals, and K. Morik (Heidelberg: Springer). doi: 10.1007/978-3-540-87481-2_21

CrossRef Full Text | Google Scholar

Satopa, V., Albrecht, J., Irwin, D., and Raghavan, B. (2011). “Finding a” kneedle” in a haystack: Detecting knee points in system behavior,” in Proceedings of the 2011 31st International Conference on Distributed Computing Systems Workshops (Piscataway, NJ: IEEE). doi: 10.1109/ICDCSW.2011.20

CrossRef Full Text | Google Scholar

Saylor, D., Dickens, A. M., Sacktor, N., Haughey, N., Slusher, B., Pletnikov, M., et al. (2016). HIV-associated neurocognitive disorder–pathogenesis and prospects for treatment. Nat. Rev. Neurol. 12:309. doi: 10.1038/nrneurol.2016.53

CrossRef Full Text | Google Scholar

Schretlen, D. J., Testa, S. M., and Pearlson, G. D. (2018). Calibrated neuropsychological normative system™. PAR, Inc. Available online at: https://www.parinc.com/Products/Pkey/58 (accessed March 15, 2022).

Google Scholar

Strauss, E., Sherman, E. M. S., and Spreen, O. (2006). A compendium of neuropsychological tests: Administration, norms, and commentary, 3rd Edn. Oxford: Oxford University Press.

Google Scholar

Tedaldi, E. M., Minniti, N. L., and Fischer, T. (2015). HIV-associated neurocognitive disorders: The relationship of HIV infection with physical and social comorbidities. Biomed. Res. Int. 2015:641913. doi: 10.1155/2015/641913

CrossRef Full Text | Google Scholar

Vance, D. E., Fazeli, P. L., Ball, D. A., Slater, L. Z., and Ross, L. A. (2014). Cognitive functioning and driving simulator performance in middle-aged and older adults with HIV. J. Assoc. Nurses AIDS Care 25, e11–e26. doi: 10.1016/j.jana.2013.12.001

CrossRef Full Text | Google Scholar

Wadley, V. G., Okonkwo, O., Crowe, M., Vance, D. E., Elgin, J. M., Ball, K. K., et al. (2009). Mild cognitive impairment and everyday function: An investigation of driving performance. J. Geriatr. Psychiatry Neurol. 22, 87–94. doi: 10.1177/0891988708328215

CrossRef Full Text | Google Scholar

Walshe, E. A., Romer, D., Kandadai, V., and Winston, F. K. (2020). A novel health-transportation partnership paves the road for young driver safety through virtual assessment. Health Aff. 39, 1792–1798. doi: 10.1377/hlthaff.2020.00802

PubMed Abstract | CrossRef Full Text | Google Scholar

Walshe, E. A., Ward-McIntosh, C., Romer, D., and Winston, F. K. (2017). Executive function capacities, negative driving behavior and crashes in young drivers. Int. J. Environ. Res. Public Health 14:E1314. doi: 10.3390/ijerph14111314

PubMed Abstract | CrossRef Full Text | Google Scholar

Winston, F. K., Kandadai, V., and Hill, S. D. (2019). Ohio portable driver simulation system pilot: Implementation of a virtual driving test within ohio’s driver licensing workflow (research brief). Philadelphia, PA: Children’s Hospital of Philadelphia Research Institute.

Google Scholar

Woods, S. P., Rippeth, J. D., Frol, A. B., Levy, J. K., Ryan, E., Soukup, V. M., et al. (2004). Interrater reliability of clinical ratings and neurocognitive diagnoses in HIV. J. Clin. Exp. Neuropsychol. 26, 759–778. doi: 10.1080/13803390490509565

CrossRef Full Text | Google Scholar

World Health Organization (2017). Global health observatory (GHO) data–HIV/AIDS. Available online at: https://www.who.int/gho/hiv/en/ (accessed March 15, 2022).

Google Scholar

Yusuf, A. J., Hassan, A., Mamman, A. I., Muktar, H. M., Suleiman, A. M., and Baiyewu, O. (2017). Prevalence of HIV-associated neurocognitive disorder (HAND) among Patients attending a tertiary health facility in Northern Nigeria. J. Int. Assoc. Provid. AIDS Care 16, 48–55. doi: 10.1177/2325957414553839

PubMed Abstract | CrossRef Full Text | Google Scholar