Stefan C. Michalski
Caroline Ellison
Ancret Szpak
Tobias Loetscher- UniSA Justice & Society, University of South Australia, Adelaide, SA, Australia
People with neurodevelopmental disorders are often considered unsuitable or incapable of working in open employment. When employment is available, tasks are often limited, and opportunities for career development are restricted. Policy and funding constraints leave people with disabilities without an opportunity to develop skills due to the additional time and costs for employers. To overcome these barriers, virtual environments have been proposed as a safe and reliable solution for training. An important prerequisite for a wider uptake of training in virtual environments are demonstrations that the training leads to improved performance in the real world. This is particularly true for people with neurodevelopmental disorders, as transferring learnings from one context to another can be challenging. A systematic review was conducted to assess whether training in virtual environments can be used to improve real-world vocational skills in people with neurodevelopmental disorders. After a systematic search in six databases, eight out of the initially identified 1,806 articles met the inclusion criteria. The findings from these eight studies demonstrate that people with neurodevelopmental disorders can transfer vocational skills from virtual environments to real-world settings. With substantial technological improvements, a surge in accessibility, and improved affordability, there is a need to build upon the promising results identified in this review.
Introduction
Rates of people with neurodevelopmental disorders in competitive (i.e., open) employment are disproportionately low (Waghorn et al., 2012; Kinoshita et al., 2013; Modini et al., 2016b; Bush and Tasse, 2017). People with neurodevelopmental disorders such as Autism spectrum disorder and intellectual disability encounter considerable challenges in obtaining and sustaining work in open employment. Employment provides numerous benefits, including structure, a source of social support, opportunities to make decisions, income and greater independence (Modini et al., 2016b). Thus, despite frequent challenges such as stigmatising views and a low priority given by employers, it is evident that most still want to work (Kinoshita et al., 2013; Modini et al., 2016a). While open employment is the goal for many, the most common outcome is employment in sheltered workshops or unemployment (Lehman, 1995; Colella and Bruyère, 2011; Cimera et al., 2012; Winsor et al., 2018).
Sheltered employment is a program in which people with disabilities receive training to develop work-related skills and behaviours. The underlying premise is that individuals need preparation before entering open employment; a “train then place” model. The inherent value of sheltered workshops includes tailored support, longstanding social relationships and the opportunity to work. While sheltered workshops may be appropriate for many individuals, the tasks available are often limited, repetitive, and career development opportunities can be restricted. Cimera (2011) suggests only a few individuals transition out of sheltered workshops into open employment. Resource constraints and a lack of innovation are real problems faced by transition educators, making it difficult for people with disabilities to achieve their goals in open employment (Walker et al., 2019). Perhaps if sheltered workshops had access to more flexible, affordable and advanced training tools, they would achieve greater success in transitioning their clients into competitive positions.
Supported employment is an alternative approach where people with disabilities are placed in open employment, often without extensive preparation. The premise is that individuals are placed into competitive positions while receiving intensive on-the-job support; a “place then train” model. Supported employment is not dissimilar to in-vivo training, where learners are immediately engaged in a targeted activity in the same location where it is performed in real life. In-vivo is generally the preferred method of training as it exposes learners to the work setting and avoids the need to transfer learnings from artificial environments, like in computer-based training. Ideally, people with neurodevelopmental disorders would participate in supported employment all the time, although there are some drawbacks (Choi et al., 2012; Panerai et al., 2018; Lindsay and Lamptey, 2019).
The effectiveness of supported employment is well-established, but challenges remain, which prevent its wide-spread implementation (Marino and Dixon, 2014; Chen et al., 2015). One drawback is concerns surrounding its cost-effectiveness. As trained staff and focused interventions are required, it can become a time and resource-intensive process (Rogers, 2000; Parsons and Cobb, 2011; Marino and Dixon, 2014). Salkever (2013) reviewed the most common supported employment practise (Individual Placement and Support) and found its wide-spread expansion did not support long-term financial gain for employers. In essence, ongoing assistance effectively helps people with disabilities in competitive positions but it is not cost-effective to maintain (Vornholt et al., 2018). Educators and researchers have considered virtual environments to be a cost-effective method to help people in learning environments (Cromby et al., 1996; Bozgeyikli et al., 2014; Smith et al., 2020).
Virtual environments have appealed to educators as a way to bring the real world into the classroom (van Vonderen, 2004; Newbutt et al., 2016b). Immersive and interactive virtual environments may benefit people with disabilities via simulation of training which permits practising as if in real life. Real-world scenarios can be created to provide safe, repeatable and targeted training that focuses on improving the social and practical skills needed in the real world. Various environments and scenarios can be simulated to create realistic interactions for users to practise specific skills and develop confidence before attempting tasks in real-world settings, such as in open employment. Generally, people with intellectual disabilities and Autism spectrum disorder have limited opportunities for skill development in the real world due to concerns surrounding safety and ability (Dempsey and Ford, 2009). Certain life skills make sense for training to occur in-vivo; where one will perform the task in real life. Though, for training work-related tasks such as job interviews and the use of hazardous equipment, perhaps there is merit to training in virtual environments (Volkmar et al., 2014).
Virtual environments can be manipulated in ways that take advantage of the learning styles of people with neurodevelopmental disorders (Kandalaft et al., 2013; Knight et al., 2013; Newbutt, 2013; Newbutt et al., 2016a). Features can be enhanced or diminished, objects can be isolated, highlighted or removed, and relationships and associations can be emphasised. The environment can be controlled and manipulated in unique ways that make it particularly interesting for educators (van Vonderen, 2004). Tailored prompts and corrective feedback can be seamlessly integrated into training to reinforce teachings (Kandalaft et al., 2013; Knight et al., 2013; Newbutt, 2013; Newbutt et al., 2016b). Furthermore, visual and auditory cues have been used to help initiate responses and memorise steps in a procedure (Lancioni et al., 1999; Riffel et al., 2005; Mechling, 2007; Sauer et al., 2010; Bailey et al., 2011).
There is encouraging evidence that virtual environments can be used to improve life skills and social skills in people with neurodevelopmental disorders (Standen and Brown, 2005; Ramdoss et al., 2011, 2012). The two most common forms of training using virtual environments for people with disabilities are video-based and computer-based interventions. Video-based interventions involve the learner observing a target skill via video, and then opportunities are provided for the person to imitate the behaviours shown (Ramdoss et al., 2012). There is encouraging evidence to support both video-based interventions (Norman et al., 2001; Rehfeldt et al., 2003; Sturmey, 2003; Mechling et al., 2005; Sigafoos et al., 2005; Rayner et al., 2009; Mechling and O'Brien, 2010; Munandar et al., 2020) and computer-based interventions (Davies et al., 2003; Hutcherson et al., 2004; Hansen and Morgan, 2008; Ayres et al., 2009). However, unlike video-based interventions, computer-based interventions allow the learner to interact via external hardware (e.g., touch screen, keyboard, mouse, joystick) (Mechling and Gast, 2003; Ramdoss et al., 2012). Some view this interaction with the learner as a way of actively involving the participant, which may be beneficial for learning outcomes (Mechling et al., 2005).
The purpose of training in virtual environments is to develop valuable and functional skills to apply in the real world, such as obtaining and sustaining employment. Though, a significant challenge for people with neurodevelopmental disorders that is widely reported in the literature is difficulties in generalising learned skills from one environment to another (Hwang and Hughes, 2000; Rogers, 2000; Parsons and Mitchell, 2002; Westwood, 2009; Ramdoss et al., 2012; Knight et al., 2013). People with neurodevelopmental disorders tend to have difficulty applying learned behaviours in new tasks or contexts, such as in spontaneous situations. Thus, it is unclear how well the skills learned in virtual environments can be applied to real-world settings.
An essential prerequisite for a wider uptake of training in virtual environments are demonstrations that the training leads to improved performance in the real world. That is, the trained skills transfer to the real-world setting. In this paper, transfer has been defined as the process by which skills, abilities and knowledge developed through training are applied in a real-world situation or task (Baldwin and Ford, 1988). It is insufficient to assess the effectiveness of virtual environments by only quantifying the extent of improvement in training. This is because the results can almost always be expected to be positive, mainly due to practise effects (Michalski et al., 2019). Measures of real-world performance are therefore needed to determine the added value of training.
The potential benefit of training in virtual environments is clear, but its effectiveness in this population is less certain. Thus, it is necessary to assess virtual environments' effectiveness to improve work-related skills by reviewing articles that measure real-world transfer. This review's primary aim is to synthesise the evidence of virtual environments as a tool to train vocational skills in people with neurodevelopmental disorders.
Method
Search Strategy
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) were followed throughout the review process (Liberati et al., 2009). The review was not pre-registered. A systematic literature search was performed on August 29, 2019. The search was updated on February 8, 2021. The following databases were used: PsycINFO, Ovid Nursing Database, Ovid MEDLINE, Ovid Emcare, and Embase. Psychology, medical and health databases were accessed to cover the scope of this current review. In addition, IEEE Xplore was searched on May 3, 2021. For all mentioned databases, the following search was conducted: [(“virtual reality” OR “virtual environment*” OR “augmented reality” OR “mixed reality”) AND (vocation* OR job OR work OR employment OR workplace OR profession* OR occupation) AND (disab* OR mental health OR “intellectual disability” OR “developmental disability” OR “learning disability” OR “brain injury” OR Autism OR retardation OR stroke OR schizophrenia OR bipolar OR depress* OR ADHD OR dyslexia)].
Inclusion Criteria
Peer-reviewed articles and conference papers published in English were included if they met the inclusion criteria while reviews and meta-analyses were not considered. Each article must have included (a) participants with a neurodevelopmental disorder; (b) a virtual environment; (c) vocational theme; and (d) real-world transfer. Our definition for each factor is detailed in turn.
a) The Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) includes a category of conditions referred to as the “neurodevelopmental disorders.” Studies with at least half of the participants diagnosed with a neurodevelopmental disorder were included.
b) To have been considered a virtual environment in this review it must have included the following components: (i) display or projection of an image; (ii) ability to interact in the environment; (iii) provision of sensory feedback (e.g., visual, auditory or haptic) (Gray, 2017; Michalski et al., 2019).
c) For this current review, a study was considered to have a vocational theme if the study outcome was work/job focused.
d) A study was considered to measure real-world transfer if real-world performance was measured pre- and post-training.
Article Selection
Four reviewers completed the article selection and screening process using Covidence systematic review management software (Covidence, 2020). Titles and abstracts were screened to identify studies that appeared eligible for inclusion. Full texts were sourced and read for articles that appeared eligible or for which eligibility could not be determined. During the full-text reading, articles were included if they met the inclusion criteria. At least two reviewers screened and read each full-text article. Articles were only included when both reviewers agreed. If a conflict arose in any stage during the article selection and screening process, the reviewers resolved the dispute via discussion until a consensus was reached. Reasons for all excluded articles during full-text screening are listed in Figure 1. Additionally, reference lists of included articles were scanned for additional articles.
Figure 1. The article selection and screening process using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram (Liberati et al., 2009).
Data Extraction
Data extracted from selected studies included information regarding sample, diagnoses, intervention, real-world performance and employment outcomes post-study. Only measures of real-world performance were extracted. Only data from participants with neurodevelopmental disorders were reported. Humm et al. (2014) was excluded as results from participants with Autism could not be separated from participants with other diagnoses.
Critical Appraisal
A critical appraisal for quasi-experimental studies was used to assess the methodological quality and risk of bias (Tufanaru et al., 2020). Items in the appraisal cover the presence or absence of basic elements of a study, including appropriate statistical analyses, a control group, reliability of measures and group similarity.
Two authors assessed each study. Both authors provided yes, no, unclear or not applicable responses to each of the nine items in the appraisal. Conflicts were resolved until a consensus was reached. The initial agreement was 90.5% before the two raters resolved conflicts.
Yes responses indicate a satisfactory level of quality was reached. All responses were tallied, and yes responses were reported as a percentage. Not applicable responses (8%) were not included in the analysis. Higher yes scores index greater methodological quality with the maximum being 100%. No studies were excluded based on the critical appraisal.
Results
Search Results
A total of 1,806 papers were identified through database searching and a snowballing strategy designed to identify additional articles. These publications were imported into EndNote where 647 duplicates were removed. Titles and abstracts were screened for the remaining 1,159 papers. Fifty-five articles appeared eligible for inclusion and were selected for full-text review. Forty-seven articles were excluded with the reasons outlined in Figure 1. Eight studies were included in this systematic review.
Critical Appraisal
The methodological quality of the included studies varied considerably (37.5–100%), with a mean satisfactory level of 82% across the eight studies.
Description of Studies
Of the eight articles, two reported multiple assessments resulting in a total of 10 real-world performance assessments. Studies compared training in virtual environments to different control groups: no training (four assessments), real-world training (one assessment), and workbook training (one assessment). Four assessments included pre-post measurements without a control group. Five assessments included participants with intellectual disabilities, three assessments included participants with Autism spectrum disorder, and two assessments included a mixed sample of participants with intellectual disability and Autism spectrum disorder.
Five studies used virtual reality job-interview training programs. These programs require users to engage in conversations with a virtual training agent. While each program is unique, they were similar in that discussion topics varied between sessions, users had freedom in their responses, and the virtual trainers displayed numerous emotions and feedback based on the user's response. Brooks et al. (2002) trained participants to prepare simple meals and identify hazards in a computer-based simulation of a virtual kitchen. Chang et al. (2013) trained meal preparation on a real table with the assistance of a flat-screen display used to provide cues and instructions. Mendozzi et al. (2000) trained simple warehouse tasks, including selecting objects by clicking on them with a mouse and moving them to another location. Mendozzi et al. (2000) also trained workshop skills which included the assembly of a torch, where a flat-screen display was used to provide textual hints to help participants follow the procedure.
Main Findings
Training in virtual environments significantly improved real-world performance from pre-test to post-test in nine out of 10 real-world assessments. All assessments with no-training controls (four) found training in virtual environments to be significantly more effective than no training. All assessments with real-world training controls (two) found no significant differences in comparison to training in virtual environments. One out of two assessments with a workbook training control found training in virtual environments to be significantly more effective, the other found no significant difference. See Table 1 for a summary of the results.
Table 1. Summary of included articles.
Discussion
This systematic review demonstrates that people with neurodevelopmental disorders can improve vocational skills via training in virtual environments. In nine out of 10 performance assessments, training in virtual environments lead to a significant improvement from pre-test to post-test in real-world settings. Importantly, the findings show that virtual training is more effective than no training. Four assessments with no-training controls found training in virtual environments to be significantly more effective. All available evidence suggests that training in virtual environments can enhance real-world performance in comparison to no training. These findings demonstrate the potential of virtual environments as a complementary tool for training.
No-training controls are essential for determining the added benefit of training but tell us little about its usefulness in comparison to other forms of training. One out of four assessments with an active control group found training in virtual environments to be significantly more effective. It is important to note that non-significant differences were found in the remaining three assessments. Overall, these findings are quite encouraging and suggest that training in virtual environments may be comparable to the outcomes of real-world training.
The findings from this review particularly highlight the effectiveness of conversational virtual human agents to develop job-interview skills. All studies assessing job interview skills found role-play performance improved after training in virtual environments. By practising speech and having the chance to go over their responses, users were able to improve content and delivery skills (Strickland et al., 2013; Walker et al., 2019). Strickland and colleagues (Strickland et al., 2013) found that participants were better able to improve content skills (i.e., verbal skills) in comparison to than delivery skills (i.e., non-verbal skills). However, the studies included were not designed to improve these delivery skills, perhaps due to the limitations in technology at the time. Modern technology may be better equipped to target non-verbal skills in training. For example, posture can be assessed via body sensors, facial expressions can be examined via video recordings, and eye contact can be measured via eye-tracking (Rogers et al., 2018). Thus, while virtual environments in this review were not accommodating for non-verbal skills, future training simulations could target these skills.
Assessments of real-world performance may identify whether participants improved their skills, but this provides no insight to participants' future employment prospects. One of the included studies (Smith et al., 2014) provided follow-up information regarding participants success in finding a job at a 6-month follow up (Smith et al., 2015b). Interestingly, the participants that trained in virtual environments were more likely to be competitively employed in comparison to participants that received no training (Smith et al., 2015b). Similar trends in employment have been found at five and 6-month follow-ups among other clinical populations (e.g., Autism spectrum disorder, bipolar disorder, major depressive disorder, post-traumatic stress disorder, and schizophrenia) (Humm et al., 2014; Smith et al., 2015a,c; Smith et al., 2017). The evidence that training in virtual environments leads to real opportunities is promising, though more studies providing follow-up data are needed to support this finding.
A job interview is a confronting experience and a critical part of obtaining a job in open employment, thus it is essential to teach people the skills needed for an interview. While this is an important step, employers look for individuals which have the necessary skills required for the job (Hall and Wilton, 2011). Three studies investigated skill transfer in applied tasks and found positive results. Mendozzi et al. (2000) assessed skill transfer from virtual environments in an item retrieval and assembly task. Furthermore, Brooks et al. (2002) evaluated the efficacy of using a virtual kitchen for vocational training in food preparation and hazard awareness training scenarios. Finally, Chang et al. (2013) assessed food preparation skills in participants with cognitive impairments. In taking the next steps, more researchers should continue to target skills that people will need on the job. Having an ability to practise skills in safe and repeatable settings may benefit certain training aspects, including procedural learning and adaptation to different work settings.
Training in virtual environments puts the learner in control of the learning situation but having an instructor present might be ideal for providing support in the learning process. In a study by Walker and colleagues (Walker et al., 2016), training job-interview skills in virtual environments was combined with immediate face-to-face coaching. Coaching included a reflection on the participant's training performance and strategies on how to improve future responses. This was designed to promote the generalisation of skills in real-world settings. Ideally, learners' practise with the support of a therapist or caregiver all the time; however, we know one-on-one training is expensive and time-consuming. Perhaps an integration of training in virtual environments with the guidance of an instructor is a practical yet effective approach. If training in virtual environments is to be adopted by organisations, understanding how training should be structured is essential. Information on the dose and frequency of training is therefore critical, yet only a few studies included in this review reported the amount of time spent training.
The available studies training vocational skills in virtual environments have found encouraging results despite using technology that would now be considered outdated. In the eight included studies, three were computer-based, using keyboard, mouse, and joystick interactions. While most studies found positive skill transfer with such interactions, Mendozzi et al. (2000) found mixed results. They found their participants improved in an item retrieval task but did not improve in an item assembly task. These mixed results could be attributed to the technology used. In the study, participants were able to benefit from training in a virtual environment in the item retrieval task, which, by nature, requires fewer complex interactions. However, in the item assembly task of a torch, the physical interactions in training would be considered unrealistic, as using a keyboard and mouse is quite different from assembling objects by hand. This study was published 20 years ago, and technologies are now available to provide more realistic interactions via hand tracking and haptic feedback. It stands to reason that a greater realism in the interactions would improve the learning of practical hands-on skills.
Virtual environments offering a low level of immersion were found to be useful in this review. All studies used flat screen displays in the form of televisions and desktop monitors. This technology would generally be considered to offer a low level of immersion in comparison to what is currently available on the market (e.g., wearable head-mounted displays). Encouragingly, studies in this review reported that sessions were well-attended. As no studies used highly immersive technology, it remains unclear how people with neurodevelopmental disorders may respond in more immersive experiences. Future studies that include immersive virtual environments should report participant dropout rates as this is one indicator of comfort and usability.
Before immersive virtual environments can be encouraged for training people with neurodevelopmental disorders, information regarding adverse effects is critical and must be carefully considered. Self-report measures provide insight on a human level (i.e., how the participant is thinking and feeling), though such findings must be interpreted with caution. There are issues around assessing cybersickness via self-report, and these issues may be confounded in this population. People with neurodevelopmental disorders generally have communication barriers and a tendency to positively self-report (Schwartz and Rabinovitz, 2003; Cummins, 2005). Careful investigation of these responses must be considered to ensure the safe use of virtual reality and reduce the risk of adverse symptoms.
Conclusion
This review demonstrates that training in virtual environments can be used to improve vocational skills in people with neurodevelopmental disorders. While these results are encouraging, they are only supported by a small evidence base and a very limited range of skills assessed. The authors note 20 articles did not meet the inclusion criteria due to lack of a real-world transfer assessment. There seems a real lack in the translation from prototype to real-world testing. Many studies included in this review used technologies that would be considered outdated today. With substantial technological improvements, a surge in accessibility, and improved affordability, there is a need to build upon the promising results identified in this review. Support organisations may benefit by using virtual environments to boost productivity in the tasks people with disabilities have been commonly assigned to. However, the real advantage is providing people with opportunities to fulfil their goals and ambitions through safe virtual tasks that simulate activities they are typically excluded from in real life.
Data Availability Statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Author Contributions
SM, CE, AS, and TL conception of the work, article selection, and screening. SM wrote the manuscript. All authors revised the work critically for intellectual content and have read and approved the manuscript.
Funding
SM was supported by the Australian Government Research Training Program Scholarship and the Channel 7 Children's Research Foundation. TL was supported by the National Health and Medical Research Council (NHMRC) Dementia Research Leadership Fellowships (GNT1136269).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Ayres, K. M., Maguire, A., and McClimon, D. (2009). Acquisition and generalization of chained tasks taught with computer based video instruction to children with autism. Educ. Train. Dev. Disabil. 44, 493–508.
Bailey, R., Willner, P., and Dymond, S. (2011). A visual aid to decision-making for people with intellectual disabilities. Res. Dev. Disabil. 32, 37–46. doi: 10.1016/j.ridd.2010.08.008
Baldwin, T. T., and Ford, J. K. (1988). Transfer of training: a review and directions for future research. Pers. Psychol. 41, 63–105. doi: 10.1111/j.1744-6570.1988.tb00632.x
Bozgeyikli, L., Bozgeyikli, E., Clevenger, M., Gong, S., Raij, A., Alqasemi, R., et al. (2014). “VR4VR: towards vocational rehabilitation of individuals with disabilities in immersive virtual reality environments,” in 2014 2nd Workshop on Virtual and Augmented Assistive Technology (VAAT) (Minneapolis, MN).
Brooks, B., Rose, F., Attree, E., and Elliot-Square, A. (2002). An evaluation of the efficacy of training people with learning disabilities in a virtual environment. Disabil. Rehabil. 24, 622–626. doi: 10.1080/09638280110111397
Burke, S. L., Bresnahan, T., Li, T., Epnere, K., Rizzo, A., Partin, M., et al. (2018). Using virtual interactive training agents (ViTA) with adults with autism and other developmental disabilities. J. Autism Dev. Disord. 48, 905–912. doi: 10.1007/s10803-017-3374-z
Burke, S. L., Li, T., Grudzien, A., and Garcia, S. (2021). Brief Report: Improving employment interview self-efficacy among adults with autism and other developmental disabilities using virtual interactive training agents (ViTA). J. Autism. Dev. Disord. 51, 741–748. doi: 10.1007/s10803-020-04571-8
Bush, K. L., and Tasse, M. J. (2017). Employment and choice-making for adults with intellectual disability, autism, and down syndrome. Res. Dev. Disabil. 65, 23–34. doi: 10.1016/j.ridd.2017.04.004
Chang, Y.-J., Kang, Y.-S., and Huang, P.-C. (2013). An augmented reality (AR)-based vocational task prompting system for people with cognitive impairments. Res. Dev. Disabil. 34, 3049–3056. doi: 10.1016/j.ridd.2013.06.026
Chen, J. L., Leader, G., Sung, C., and Leahy, M. (2015). Trends in employment for individuals with autism spectrum disorder: a review of the research literature. Rev. J. Autism Dev. Disord. 2, 115–127. doi: 10.1007/s40489-014-0041-6
Choi, K.-S., Wong, P.-K., and Chung, W.-Y. (2012). Using computer-assisted method to teach children with intellectual disabilities handwashing skills. Disabil. Rehabil. Assist. Technol. 7, 507–516. doi: 10.3109/17483107.2011.652998
Cimera, R. E. (2011). Does being in sheltered workshops improve the employment outcomes of supported employees with intellectual disabilities? J. Vocat. Rehabil. 35, 21–27. doi: 10.3233/JVR-2011-0550
Cimera, R. E., Wehman, P., West, M., and Burgess, S. (2012). Do sheltered workshops enhance employment outcomes for adults with autism spectrum disorder? Autism 16, 87–94. doi: 10.1177/1362361311408129
Colella, A., and Bruyère, S. M. (2011). Disability and employment: new directions for industrial and organizational psychology. APA Handb. Ind. Organ. Psychol. 1, 473–503. doi: 10.1037/12169-015
Covidence (2020). Covidence Systematic Review Software. Available online at: https://www.covidence.org/home (accessed May 3, 2021).
Cromby, J. J., Standen, P. J., and Brown, D. J. (1996). The potentials of virtual environments in the education and training of people with learning disabilities. J. Intellect. Disabil. Res. 40, 489–501. doi: 10.1111/j.1365-2788.1996.tb00659.x
Cummins, R. A. (2005). “The domains of life satisfaction: an attempt to order chaos,” in Citation Classics From Social Indicators Research, ed A. C. Michalos (Dordrecht: Springer), 559–584.
Davies, D. K., Stock, S. E., and Wehmeyer, M. L. (2003). Application of computer simulation to teach ATM access to individuals with intellectual disabilities. Educ. Train. Dev. Disabil. 38, 451–456.
Dempsey, I., and Ford, J. (2009). Employment for people with intellectual disability in Australia and the United Kingdom. J. Disabil. Policy Stud. 19, 233–243. doi: 10.1177/1044207308314946
Gray, R. (2017). Transfer of training from virtual to real baseball batting. Front. Psychol. 8:2183. doi: 10.3389/fpsyg.2017.02183
Hall, E., and Wilton, R. (2011). Alternative spaces of ‘work’ and inclusion for disabled people. Disabil. Soc. 26, 867–880. doi: 10.1080/09687599.2011.618742
Hansen, D. L., and Morgan, R. L. (2008). Teaching grocery store purchasing skills to students with intellectual disabilities using a computer-based instruction program. Educ. Train. Dev. Disabil. 43, 431–442.
Humm, L. B., Olsen, D., Be, M., Fleming, M., and Smith, M. (2014). Simulated job interview improves skills for adults with serious mental illnesses. Annu. Rev. CyberTherapy Telemed. 12, 50–54.
Hutcherson, K., Langone, J., Ayres, K., and Clees, T. (2004). Computer assisted instruction to teach item selection in grocery stores: an assessment of acquisition and generalization. J. Spec. Educ. Technol. 19, 33–42. doi: 10.1177/016264340401900404
Hwang, B., and Hughes, C. (2000). The effects of social interactive training on early social communicative skills of children with autism. J. Autism Dev. Disord. 30:13. doi: 10.1023/A:1005579317085
Kandalaft, M. R., Didehbani, N., Krawczyk, D. C., Allen, T. T., and Chapman, S. B. (2013). Virtual reality social cognition training for young adults with high-functioning autism. J. Autism Dev. Disord. 43, 34–44. doi: 10.1007/s10803-012-1544-6
Kinoshita, Y., Furukawa, T. A., Kinoshita, K., Honyashiki, M., Omori, I. M., Marshall, M., et al. (2013). Supported employment for adults with severe mental illness. Cochrane Database Syst. Rev. 2013:CD008297. doi: 10.1002/14651858.CD008297.pub2
Knight, V., McKissick, B. R., and Saunders, A. (2013). A review of technology-based interventions to teach academic skills to students with autism spectrum disorder. J. Autism Dev. Disord. 43, 2628–2648. doi: 10.1007/s10803-013-1814-y
Lancioni, G. E., den Hof, E. V., Furniss, F., O'Reilly, M. F., and Cunha, B. (1999). Evaluation of a computer-aided system providing pictorial task instructions and prompts to people with severe intellectual disability. J. Intellect. Disabil. Res. 43, 61–66. doi: 10.1046/j.1365-2788.1999.43120165.x
Lehman, A. (1995). Vocational rehabilitation in schizophrenia. Schizophr. Bull. 21, 645–656. doi: 10.1093/schbul/21.4.645
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 6:e1000100. doi: 10.1371/journal.pmed.1000100
Lindsay, S., and Lamptey, D.-L. (2019). Pedestrian navigation and public transit training interventions for youth with disabilities: a systematic review. Disabil. Rehabil. 41, 2607–2621. doi: 10.1080/09638288.2018.1471165
Marino, L. A., and Dixon, L. B. (2014). An update on supported employment for people with severe mental illness. Curr. Opin. Psychiatry 27, 210–215. doi: 10.1097/YCO.0000000000000058
Mechling, L., and O'Brien, E. (2010). Computer-based video instruction to teach students with intellectual disabilities to use public bus transportation. Educ. Train. Autism Dev. Disabil. 45, 230–241.
Mechling, L. C. (2007). Assistive technology as a self-management tool for prompting students with intellectual disabilities to initiate and complete daily tasks: a literature review. Educ. Train. Dev. Disabil. 42, 252–269.
Mechling, L. C., and Gast, D. L. (2003). Multi-media instruction to teach grocery word associations and store location: a study of generalization. Educ. Train. Dev. Disabil. 38, 62–76.
Mechling, L. C., Pridgen, L. S., and Cronin, B. A. (2005). Computer-based video instruction to teach students with intellectual disabilities to verbally respond to questions and make purchases in fast food restaurants. Educ. Train. Dev. Disabil. 40, 47–59.
Mendozzi, L., Pugnetti, L., Barbieri, E., Attree, E. A., Rose, F. D., Moro, W., et al. (2000). “VIRT–factory trainer project. A generic productive process to train persons with disabilities,” in The 3rd International Conference on Disability, Virtual Reality and Associated Technologies (Alghero).
Michalski, S. C., Szpak, A., and Loetscher, T. (2019). Using virtual environments to improve real-world motor skills in sports: a systematic review. Front. Psychol. 10:2159. doi: 10.3389/fpsyg.2019.02159
Modini, M., Joyce, S., Mykletun, A., Christensen, H., Bryant, R. A., Mitchell, P. B., et al. (2016a). The mental health benefits of employment: results of a systematic meta-review. Australas. Psychiatry 24, 331–336. doi: 10.1177/1039856215618523
Modini, M., Tan, L., Brinchmann, B., Wang, M.-J., Killackey, E., Glozier, N., et al. (2016b). Supported employment for people with severe mental illness: systematic review and meta-analysis of the international evidence. Br. J. Psychiatry 209, 14–22. doi: 10.1192/bjp.bp.115.165092
Munandar, V., Morningstar, M. E., and Carlson, S. R. (2020). A systematic literature review of video-based interventions to improve integrated competitive employment skills among youth and adults with autism spectrum disorder. J. Vocat. Rehabil. 53, 29–41. doi: 10.3233/JVR-201083
Newbutt, N. (2013). Exploring Communication and Representation of the Self in a Virtual World by Young People with Autism. Dublin: University College Dublin.
Newbutt, N., Sung, C., Kuo, H.-J., Leahy, M. J., Lin, C.-C., and Tong, B. (2016b). Brief report: a pilot study of the use of a virtual reality headset in autism populations. J. Autism Dev. Disord. 46, 3166–3176. doi: 10.1007/s10803-016-2830-5
Newbutt, N., Sung, C., Kuo, H. J., and Leahy, M. J. (2016a). The potential of virtual reality technologies to support people with an autism condition: a case study of acceptance, presence and negative effects. Annu. Rev. CyberTherapy Telemed. 14, 149–154.
Norman, J. M., Collins, B. C., and Schuster, J. W. (2001). Using an instructional package including video technology to teach self-help skills to elementary students with mental disabilities. J. Spec. Educ. Technol. 16, 5–18. doi: 10.1177/016264340101600301
Panerai, S., Catania, V., Rundo, F., and Ferri, R. (2018). Remote home-based virtual training of functional living skills for adolescents and young adults with intellectual disability: feasibility and preliminary results. Front. Psychol. 9:1730. doi: 10.3389/fpsyg.2018.01730
Parsons, S., and Cobb, S. (2011). State-of-the-art of virtual reality technologies for children on the autism spectrum. Eur. J. Spec. Needs Educ. 26, 355–366. doi: 10.1080/08856257.2011.593831
Parsons, S., and Mitchell, P. (2002). The potential of virtual reality in social skills training for people with autistic spectrum disorders: autism, social skills, and virtual reality. J. Intellect. Disabil. Res. 46, 430–443. doi: 10.1046/j.1365-2788.2002.00425.x
Ramdoss, S., Lang, R., Fragale, C., Britt, C., O'Reilly, M., Sigafoos, J., et al. (2012). Use of computer-based interventions to promote daily living skills in individuals with intellectual disabilities: a systematic review. J. Dev. Phys. Disabil. 24, 197–215. doi: 10.1007/s10882-011-9259-8
Ramdoss, S., Lang, R., Mulloy, A., Franco, J., O'Reilly, M., Didden, R., et al. (2011). Use of computer-based interventions to teach communication skills to children with autism spectrum disorders: a systematic review. J. Behav. Educ. 20, 55–76. doi: 10.1007/s10864-010-9112-7
Rayner, C., Denholm, C., and Sigafoos, J. (2009). Video-based intervention for individuals with autism: key questions that remain unanswered. Res. Autism Spectr. Disord. 3, 291–303. doi: 10.1016/j.rasd.2008.09.001
Rehfeldt, R. A., Dahman, D., Young, A., Cherry, H., and Davis, P. (2003). Teaching a simple meal preparation skill to adults with moderate and severe mental retardation using video modeling. Behav. Interv. 18, 209–218. doi: 10.1002/bin.139
Riffel, L. A., Wehmeyer, M. L., Turnbull, A. P., Lattimore, J., Davies, D., Stock, S., et al. (2005). Promoting independent performance of transition-related tasks using a palmtop PC-based self-directed visual and auditory prompting system. J. Spec. Educ. Technol. 20, 5–14. doi: 10.1177/016264340502000201
Rogers, S. J. (2000). Interventions That Facilitate Socialization in Children with Autism. J. Autism Dev. Disord. 30, 399–409. doi: 10.1023/a:1005543321840
Rogers, S. L., Speelman, C. P., Guidetti, O., and Longmuir, M. (2018). Using dual eye tracking to uncover personal gaze patterns during social interaction. Sci. Rep. 8, 1–9. doi: 10.1038/s41598-018-22726-7
Salkever, D. (2013). Social costs of expanding access to evidence-based supported employment: concepts and interpretive review of evidence. Psychiatr. Serv. 64, 111–119. doi: 10.1176/appi.ps.201100499
Sauer, A. L., Parks, A., and Heyn, P. C. (2010). Assistive technology effects on the employment outcomes for people with cognitive disabilities: a systematic review. Disabil. Rehabil. Assist. Technol. 5, 377–391. doi: 10.3109/17483101003746360
Schwartz, C., and Rabinovitz, S. (2003). Life satisfaction of people with intellectual disability living in community residences: perceptions of the residents, their parents and staff members. J. Intellect. Disabil. Res. 47, 75–84. doi: 10.1046/j.1365-2788.2003.00436.x
Sigafoos, J., O'Reilly, M., Cannella, H., Upadhyaya, M., Edrisinha, C., Lancioni, G. E., et al. (2005). Computer-presented video prompting for teaching microwave oven use to three adults with developmental disabilities. J. Behav. Educ. 14, 189–201. doi: 10.1007/s10864-005-6297-2
Smith, M. J., Fleming, M. F., Wright, M. A., Jordan, N., Humm, L. B., Olsen, D., et al. (2015a). Job offers to individuals with severe mental illness after participation in virtual reality job interview training. Psychiatr. Serv. 66, 1173–1179. doi: 10.1176/appi.ps.201400504
Smith, M. J., Fleming, M. F., Wright, M. A., Losh, M., Humm, L. B., Olsen, D., et al. (2015b). Brief report: vocational outcomes for young adults with autism spectrum disorders at six months after virtual reality job interview training. J. Autism Dev. Disord. 45, 3364–3369. doi: 10.1007/s10803-015-2470-1
Smith, M. J., Fleming, M. F., Wright, M. A., Roberts, A. G., Humm, L. B., Olsen, D., et al. (2015c). Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment. Schizophr. Res. 166, 86–91. doi: 10.1016/j.schres.2015.05.022
Smith, M. J., Ginger, E. J., Wright, K., Wright, M. A., Taylor, J. L., Humm, L. B., et al. (2014). Virtual reality job interview training in adults with autism spectrum disorder. J. Autism Dev. Disord. 44, 2450–2463. doi: 10.1007/s10803-014-2113-y
Smith, M. J., Graham, A. K., Sax, R., Spencer, E. S., Razzano, L. A., Smith, J. D., et al. (2020). Costs of preparing to implement a virtual reality job interview training programme in a community mental health agency: a budget impact analysis. J. Eval. Clin. Pract. 26, 1188–1195. doi: 10.1111/jep.13292
Smith, M. J., Smith, J. D., Fleming, M. F., Jordan, N., Brown, C. H., Humm, L., et al. (2017). Mechanism of action for obtaining job offers with virtual reality job interview training. Psychiatr. Serv. 68, 747–750. doi: 10.1176/appi.ps.201600217
Standen, P. J., and Brown, D. J. (2005). Virtual reality in the rehabilitation of people with intellectual disabilities: review. CyberPsychol. Behav. 8, 272–282. doi: 10.1089/cpb.2005.8.272
Strickland, D. C., Coles, C. D., and Southern, L. B. (2013). JobTIPS: a transition to employment program for individuals with autism spectrum disorders. J. Autism Dev. Disord. 43, 2472–2483. doi: 10.1007/s10803-013-1800-4
Sturmey, P. (2003). Video technology and persons with autism and other developmental disabilities. J. Posit. Behav. Interv. 5, 3–4. doi: 10.1177/10983007030050010401
Tufanaru, C., Munn, Z., Aromataris, E., Campbell, J., and Hopp, L. (2020). “Chapter 3: systematic reviews of effectiveness,” in JBI Manual for Evidence Synthesis, eds E. Aromataris and Z. Munn (Adelaide SA: JBI). Available online at: https://synthesismanual.jbi.global (accessed May 3, 2021).
van Vonderen, A. (2004). Effectiveness of immediate verbal feedback on trainer behaviour during communication training with individuals with intellectual disability. J. Intellect. Disabil. Res. 48, 245–251. doi: 10.1111/j.1365-2788.2003.00555.x
Volkmar, F. R., Reichow, B., and McPartland, J. C. (2014). Adolescents and Adults with Autism Spectrum Disorders. Berlin/Heidelberg: Springer Science and Business.
Vornholt, K., Villotti, P., Muschalla, B., Bauer, J., Colella, A., Zijlstra, F., et al. (2018). Disability and employment—overview and highlights. Eur. J. Work Organ. Psychol. 27, 40–55. doi: 10.1080/1359432X.2017.1387536
Waghorn, G., Saha, S., Harvey, C., Morgan, V. A., Waterreus, A., Bush, R., et al. (2012). ‘Earning and learning’ in those with psychotic disorders: the second Australian national survey of psychosis. Aust. N. Z. J. Psychiatry 46, 774–785. doi: 10.1177/0004867412452015
Walker, Z., Lee, S. J. E., Wienke, W., and Tan, D. S. (2019). A review of interview preparation via virtual and mixed reality for individuals with intellectual and developmental disorder. J. Vocat. Rehabil. 51, 87–97. doi: 10.3233/JVR-191028
Walker, Z., Vasquez, E., and Wienke, W. (2016). The impact of simulated interviews for individuals with intellectual disability. Educ. Technol. Soc. 19, 76–88.
Westwood, P. S. (2009). What Teachers Need to Know About Students With Disabilities. Camberwell: ACER Press.
Winsor, J., Timmons, J., Butterworth, J., Migliore, A., Domin, D., Zalewska, A., et al. (2018). StateData: The National Report on Employment Services and Outcomes. Boston, MA: University of Massachusetts Boston, Institute for Community Inclusion. Available online at: https://scholarworks.umb.edu/thinkwork/47/
Keywords: neurodevelopmental disorder, intellectual disability, Autism, virtual environment, virtual reality, vocational training, work, transfer
Citation: Michalski SC, Ellison C, Szpak A and Loetscher T (2021) Vocational Training in Virtual Environments for People With Neurodevelopmental Disorders: A Systematic Review. Front. Psychol. 12:627301. doi: 10.3389/fpsyg.2021.627301
Received: 09 November 2020; Accepted: 08 June 2021;
Published: 07 July 2021.
Edited by:
Giovanni Ottoboni, University of Bologna, ItalyReviewed by:
Carlos Vaz De Carvalho, Polytechnic Institute of Porto, PortugalAntónio Fernando Coelho, University of Porto, Portugal
Copyright © 2021 Michalski, Ellison, Szpak and Loetscher. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Stefan C. Michalski, stefan.michalski@mymail.unisa.edu.au