Ned T. Sahin1,2*
Rafiq Abdus-Sabur1Neha U. Keshav1
Runpeng Liu1,3
Joseph P. Salisbury1
Arshya Vahabzadeh1,4- 1Brain Power, Cambridge, MA, United States
- 2Department of Psychology, Harvard University, Cambridge, MA, United States
- 3Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, United States
- 4Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States
Background: Impairment in social communication is the primary deficit in school-aged children with autism spectrum disorder (ASD). Research has shown that there are efficacious interventions to address social communication deficits, yet their delivery is hampered by the lack of human and time resources. Emerging assistive technologies, such as smartglasses, may be able to help augment the social communication interventions currently provided by human educators and therapists. While emerging research suggests assistive socio-emotional coaching smartglasses can be effective and usable in research settings, they have yet to be studied amidst the complex social, physical, and time-constrained environment of the school classroom. This structured case study reports on the feasibility and efficacy of 16 intervention sessions of the Empowered Brain Face2Face module, a smartglasses-based social communication intervention.
Methods: A 13-year-old fully-verbal adolescent male student with a diagnosis of ASD received a total of 16 smartglasses-aided intervention sessions over a 2-week period. Interventions occurred twice-daily during school days and were facilitated by school professionals in a middle school in Massachusetts, USA. Outcomes were measured using the Social Responsiveness Scale 2 (SRS-2), a commonly used validated measure of social communication in children with ASD, by the participant's parent, paraprofessional, and two teachers. Difficulties in usability during the study were recorded through observation notes.
Results: The participant completed the 3-week study [one pre-intervention week (baseline) and two intervention weeks] without any observations of adverse effects or usability concerns. The parent and three educators completed the SRS-2 for the baseline and intervention weeks, and results demonstrated significant improvement in social communication after the intervention relative to baseline. The parent, special education teacher, and general education teacher noted marked reductions in SRS-2 total T score, with improvement in SRS-2 social communication, social motivation, social cognition, and restricted interests and repetitive behavior subscales.
Conclusion: Smartglasses are a novel assistive technology that can help facilitate social communication and behavioral coaching for students with ASD. The use of the Face2Face module by educators over a 2-week period was associated with improvements in social communication.This study supports the use of this novel technology to deliver assistive social communication and behavioral coaching in schools.
Introduction
Autism Spectrum Disorder (ASD) is a childhood onset developmental condition with a rapidly increasing prevalence, and is present in 1 in 59 school-aged children (Baio et al., 2018). Impairment in social communication is the hallmark feature of ASD, encompassing perseverative deficits in verbal and non-verbal communication (American Psychiatric Association, 2013).
Evidence suggests that social communication can be improved through a range of interventions for children with ASD (Locke et al., 2013; Watkins et al., 2017). While these efficacious interventions have been studied in the school environment, their implementation has been somewhat thwarted by a lack of educational resources as schools attempt to provide specialized educational needs to a growing number of children with ASD. The magnitude of the demand for specialized educational resources is considerable, with the US school system providing half a million children with ASD federally-mandated special education under the Individuals with Disabilities Education Act (IDEA) (Snyder et al., 2016). The mismatch between supply and demand of such educational interventions has not only led to limited support for students with ASD, but also to parental dissatisfaction (White, 2014) and burnout among school professionals (Corona et al., 2017).
Children are spending more time in school, with the length of the school day increased to ~7 h over the last few decades (Kolbe et al., 2012). With almost a third of their day being spent at school, schools have become a central part of children's lives. Schools are not only educational establishments, but highly social environments, with many interpersonal relationships and interactions between students, teachers, and other educational professionals.
Schools are fertile ground for social interventions that have been found to be broadly effective (Locke et al., 2013; Watkins et al., 2017), increase both peer and teacher interactions, and may help improve non-targeted skills such as language and inappropriate behavior (Rogers, 2000) in children with ASD. Within schools, there has been a move toward incorporating students with ASD in inclusive classrooms where they learn alongside their neurotypical peers. However, this has not resolved many of the social functioning limitations seen in students with ASD, many of whom struggle with feelings of isolation and loneliness during school time (Bauminger and Kasari, 2000). Teachers have identified many barriers to implementing successful inclusion programs, with lack of training, time, and administrative support being key factors (Werts et al., 1996).
Prompt detection of ASD and early intervention are thought to be critical to long term outcomes (Dawson et al., 2010). Interventions for social communication, especially those delivered around school-settings, are also important for long term success. As children with ASD mature, they gain greater insight into their social communication deficits. This realization has a series of consequences, including increased stress when interacting with unfamiliar peers (Lopata et al., 2008; Corbett et al., 2010), and greater anxiety in social situations where they have repeatedly been unsuccessful (Bellini, 2006; Corbett et al., 2014). While there is concern that without intervention, impairment in social functioning may be lifelong, role-play exercises and interactive games between humans can significantly increase social skills (Corbett et al., 2014).
While individuals with ASD express a desire to have a job (Hendricks, 2010), persisting social skill deficits may pose a key challenge (Hurlbutt and Chalmers, 2004). People with ASD experience high rates of unemployment/underemployment that exceed other groups with disabilities (Shattuck et al., 2012). People with ASD also often have unflattering work histories, with short-lived periods of work interspersed with long periods of unemployment (Ohl et al., 2000). While there are many barriers to people with ASD obtaining employment, it has been found that the social demands that accompany jobs are a key challenge. Social skills such as small talk (Holmes and Fillary, 2010), eye contact (Amalfitano and Kalt, 1977), emotion recognition (Kee et al., 2003), and conveyance of emotions (Zapf, 2002), have all been found to be important for jobs and job interviews. Technology-aided social skills interventions have become increasingly studied, and may provide long term benefits to people with ASD who are seeking employment (Wainer and Ingersoll, 2011; Walsh et al., 2016), however the majority target job-specific skills such as cleaning tasks (Van Laarhoven et al., 2012) or shirt folding (Bennett et al., 2013). There is an immense need to develop social skills focused interventions that aid more generalized workplace interactions (Fast, 2004).
Novel assistive technology may potentially address this disconnect between demand and availability, with the promise of improved quality of education, reduced burden on teachers, and potentially reduced costs for school districts. Socially focused interventions may be especially suitable to digitization given that they can be particularly useful for children with ASD (Locke et al., 2013), but are hampered by limited training of human providers alongside lack of physical resources (Dingfelder and Mandell, 2011; Locke et al., 2015). A range of assistive technologies have proven to be effective interventions in ASD (Bauminger-Zviely et al., 2013; Grynszpan et al., 2014), and assistive technology provides one of the most common teacher-led strategies for helping students with ASD in both general and special education classrooms (Hess et al., 2008). While technologies such as the iPad have undergone considerable research (McMurray and Pierson, 2016), newer technologies such as smartglasses have also attracted interest (Liu et al., 2017).
The Empowered Brain Technology Platform
The Empowered Brain is a tool that provides socio-emotional coaching to children and adults with ASD (Keshav et al., 2017; Liu et al., 2017). The Empowered Brain consists of a smartglasses platform in combination with a series of software modules that focus on key coaching areas, such as improving attention to social cues, helping coach facial emotion recognition, and aiding in transitioning between different environments. In this study, the Empowered Brain Face2Face module was tested on Google Glass smartglasses hardware.
Smartglasses, like Google Glass, are head-worn computerized glasses that can transmit visual and auditory information to users through a small clear optical display(s) and bone conduction/audio speaker, respectively. Smartglasses, like smartphones, typically contain a wide variety of sensors that can collect data regarding the user's body movements and interactions with the environment. These sensors include a camera, a microphone, an accelerometer, a gyroscope, and Wi-Fi/Bluetooth.
Face2Face Module
The human face is one of the most powerful tools in social communication (Jack and Schyns, 2015), and is of fundamental importance in interpersonal interactions (Pavlova et al., 2017). Faces display information regarding traits, stable features such as gender and identity, and more dynamic facial data that helps with understanding of emotion, intention, attention, and understanding speech (Lee et al., 1998; Pascalis et al., 2011).
Impairment in social cognition (Baron-Cohen, 1991; Baron-Cohen et al., 1995) and social motivation (Chevallier et al., 2012) have both been described as being relevant to the facial perception, processing, and recognition difficulties of people with ASD (Kirchner et al., 2011; Tang et al., 2015; Tanaka and Sung, 2016), including how much attention they pay to socially salient features, such as the eyes of others (Klin et al., 2002; Jones et al., 2008; Kirchner et al., 2011). Gaze indifference and gaze aversion are two proposed hypotheses that may help to explain the altered eye gaze behavior seen in ASD. In gaze indifference, the eyes of others are not seen as an important or engaging stimulus (Moriuchi et al., 2017), while in gaze aversion, eye contact is avoided as it is seen as threatening (Tottenham et al., 2014) or results in sensory overstimulation (Corden et al., 2008). These altered patterns of attention to faces and eyes may be especially pronounced during non-passive circumstances, such as interactive social situations (von dem Hagen and Bright, 2017). Certainly, given the heterogeneity of ASD in both neurobiological underpinnings and behavioral presentation, it is unlikely that one unified theory alone will explain the multitude of altered facial processing abilities or eye gaze patterns that have been described.
It is in this context that the Empowered Brain Face2Face module aims to provide a social communication intervention that improves social motivation and cognition, while simultaneously addressing the underlying challenges described by both gaze indifference and gaze aversion hypotheses. Face2Face achieves this, in part, by utilizing game-like augmented reality (AR) to increase the social motivation of the user to engage with the face of another person (Figure 1). Face2Face also provides an intervention that is graduated in intensity, difficulty, and is highly customizable through both human and artificially intelligent machine input. Face2Face relies on utilizing the relatively preserved visual skills in ASD when delivering its cues, and uses audio alerts that are considerate of the sound hypersensitivity experienced by many with ASD. The approach of Face2Face is reflective of its origins as an updated and enhanced version of Face Game, a previously described smartglasses research app that has been studied in ASD (Keshav et al., 2017; Liu et al., 2017).
Figure 1. Intervention aimed to increase face-directed gaze via an augmented-reality game on smartglasses. (A) User can start the game (Face2Face) by voice command or tapping the side trackpad of the glasses (Google Glass Explorer Edition). He sees and hears feedback via the screen and bone-conduction speaker. (B) User has a social interaction (e.g., a conversation, in the case of a verbal participant), and the game gives feedback in realtime while the pair maintain interaction. (C) Representation of on-screen feedback: the user has maintained gaze with the partner for a predefined period and earns a star and temporary face mask as reward. (D) When mutual gaze is lost, the user receives graphical, verbal, and auditory coaching to resume. (E) As the game progresses, it takes longer periods of gaze maintenance to earn further stars and masks.
To play Face2Face, the user wears the smartglasses system while the Face2Face module is running, and interacts with another person who will be positioned in front of the user. This interactive facilitation recognizes that “real-world” situations allow for the challenges of social communication to be most apparent (von dem Hagen and Bright, 2017), and reduce barriers to generalizability of learned skills (Shindorf, 2016). The Face2Face module requires two people to be present: the user and the facilitator (another person who will help facilitate the Empowered Brain-augmented interaction).
The Empowered Brain can detect the presence of a human face, determine where a user is looking relative to that face, and help guide user's gaze toward the face in real-time. Guidance to the user is provided through visual directional prompts and auditory tones that vary depending on the positioning of the user's gaze and the positioning of the detected face. The Empowered Brain modules use developmentally and contextually appropriate game-like elements to make the experience engaging and fun for both child, adolescent, and adult users. For example, a cartoon-like character is optically superimposed over the face that they are being guided toward.
This cartoon-like character gradually fades as the user moves his gaze toward the target face, eventually disappearing completely. At this point the user obtains points for successful task completion. The module has a series of levels and difficulty settings. Specific game data elements are recorded and transmitted to a secure central artificial intelligence (AI) powered processing center where they are available for viewing through a web-based dashboard. User performance is measured through graphs of rewards, events, and attention data. The socio-emotional coaching apps of the Empowered Brain have been previously found to be well-tolerated (Keshav et al., 2017), feasible to use (Liu et al., 2017), free from adverse effects in people with ASD (Sahin et al., 2018), and associated with improvements in the symptoms of ASD (Liu et al., 2017) and attention deficit hyperactivity disorder (Vahabzadeh et al., 2018). These studies were, however, conducted in controlled research settings, utilizing the caregivers of participants with ASD, and were based on a single intervention session.
The Empowered Brain features key innovations in physical design, AI, and data analytics, and is supported by software, engineering, and data partnerships with a number of technology companies including X (formerly Google X, Mountain View, CA), Affectiva (a leading Emotion AI company, Boston, MA), and Amazon (use of an experimental AI technology).
Educator-Facilitated and Classroom-Based
The real-world school setting has proven itself to be a more difficult environment to provide a social communication intervention than a controlled research setting (Lawton and Kasari, 2012). Teachers and their paraprofessional colleagues may be ideal personnel to help deliver school-based communication interventions in students with ASD (Lawton and Kasari, 2012). Their intimate knowledge of a student's strengths, weaknesses, and style of learning, combined with their established position in delivering guidance, places them in a unique position. Students' perception of teacher support and teacher-led promotion of interaction and respect have been linked to increased student motivation and engagement (Ryan and Patrick, 2001).
This report outlines the efficacy, usability, and safety of the Empowered Brain Face2Face module intervention twice during every school day over a 2-week time period. In this report, the facilitators were several of the school professionals who were familiar with the research participant (education paraprofessional, special education teacher, general education teacher). Additionally, the intervention was used during the school day, and within an inclusive classroom setting. Outcomes were measured with the Social Responsiveness Scale 2 (SRS-2) School-age Form. The SRS-2 is a validated social communication measure used in ASD populations (Constantino and Gruber, 2012), and consists of 65 items, resulting in a total score, and five subscale scores (social awareness, social cognition, social communication, social motivation, and restricted interested and repetitive behaviors). The SRS-2 can be completed by teachers (Dickson et al., 2017) and parents (Ashman et al., 2017), with recent research noting that parental ratings may be higher than that of teachers (Nelson et al., 2016).
This case study attempted to use some elements of a single-subject design, although did not include repeated measures during individual phases of the study, an important element of single-subject studies. The focus on incorporating some elements of single-subject design was deemed to be important as single-subject research can be used as a means to testing and understanding the use of novel technology in educational settings, and has been identified as an important contributor to evidence-based practice in special education (Horner et al., 2005). Single-subject research has already helped to identify computer-assisted guidance for students with ASD as an evidence-based practice (Barton et al., 2017). The lack of repeated measures during individual phases is however a key limitation.
Methods
A 3-week structured case study of the Empowered Brain Face2Face module was conducted in a 13-year-old male student with ASD (“the participant”), in a middle school in Massachusetts.
A Case Study Incorporating Elements of a Single-Case Design
This case study utilized some components of single-case experimental design. Single-case experimental approaches have been shown to be methodologically sound (Horner et al., 2005; Kratochwill and Levin, 2010; Smith, 2012), capable of demonstrating effectiveness of interventions, and is especially suitable for the assessment of outcomes of psychological/behavioral interventions (Robey et al., 1999; Borckardt et al., 2008; Smith, 2012).
Our case study incorporated the four of key features of a high-quality single-case design as far as it was possible to do so (McMillan, 2004). One of these features is using ratings or measures that are reliable and consistent. Our study uses a validated and reliable 65-item scale that is viewed as a gold standard measure for measuring social communication in people with ASD. Three other important features that have been identified, and are included in our methodology, are: (1) having a detailed description of the measurement and treatment conditions (the school classroom); (2) having at least one baseline and treatment phase; and (3) changing only one variable between phases (use of the Empowered Brain smartglasses).
The fifth and final feature of single-case design is the ability to repeat the measurement as often as necessary. In our study, the key behavioral measure was repeated three times, at the end of the baseline week, and at the end of each of the two intervention weeks. While having repeated measures within each study week would offer more data, and would more robustly fulfill this fifth feature on paper, there were a number of factors that made this impractical. Firstly, the educators in our sample had limited time to provide more regular rating assessments as they were not only the facilitators of the intervention, but also continued in their full-time teaching role throughout the study. Secondly, the use of the SRS-2 as a daily measure of social communication has not been studied, and the authors believe that other simpler behavioral measures may be more appropriate if more regular measurements were needed. Thirdly, a reasonable amount of data was collected during our study. The SRS-2 is a robust assessment of social communication, with 65 different scored items, and was performed by four different individuals on three different occasions, providing a total of 780 rated items regarding the participant. However, the lack of daily/near daily data are noted by the authors as a limitation in this study, and makes the study design most consistent with a case study.
The study commenced with a baseline week (Week 1), during which the participant received no intervention, and had his regular school and home schedule. At the end of the week, school educators and his mother completed the SRS-2 based on the interactions and behaviors they witnessed during the baseline week. The SRS-2 was completed by three school educators: the participant's special education teacher, general education teacher, and assigned paraprofessional. The use of cross-informant (educator and parental) ratings is the gold standard in child behavioral assessments (Dickson et al., 2017). The participant's baseline week was the control for this report.
The study then proceeded to the first intervention week (Week 2) where the participant received twice-daily Face2Face interventions. The intervention was facilitated by one of his school educators, during which time the participant continued to be in the classroom alongside his peers. The intervention was 10 min long and was delivered at approximately the same times each day. At the end of Week 2, following eight intervention sessions, the participant's educators and parent each completed an SRS-2 based on the behavior seen during that week. The second intervention week (Week 3) was a duplicate of the first intervention week, and a repeat SRS-2 was completed at the end of the week.
The participant completed the intervention during two class periods, one general education class and one special education class. The structure of both classes allowed for the teacher to facilitate the intervention for 10 min during the class period, within his classroom, while another education professional was in the room.
The Intervention
During the intervention weeks, the participant received the intervention twice every school day during the same school periods (Figure 2). The intervention was provided by his general education teacher in the morning, and his special education teacher in the afternoon. One of the 16 interventions was provided by the paraprofessional during a class when the special education teacher was unexpectedly unavailable. The intervention ran concurrently to the class the participant was attending. The educator identifies a peripheral location in the classroom where the intervention was performed, with the student and educator sat facing each other (Figure 3).
Figure 2. Study design. (A) Overall design of the 3-week study. A baseline week was followed by 2 weeks where Empowered Brain intervention was added to the interventions the school otherwise proveded. At the end of the baseline week and each Empowered Brain week, the participant was rated according to the gold-standard autism rating scale, the Social Responsiveness Scale-2nd Edition (SRS-2) (See Figure 4). SRS–2 ratings were conducted in all cases by the participant's general education teacher, the special education teacher, paraprofessional, and the parent. (B) Design of each day of the study during the intervention period. Two 10-min intervention sessions were run daily, with a late morning session in the participant's regularly scheduled general-education class (See Figure 3), and an afternoon session in his regularly scheduled special-education pullout class.
Figure 3. Intervention in the context of an inclusive, general-education classroom. (A) Teacher and study participant before onset of an intervention session. (B) Intervention session begins when participant puts on the smartglasses running the Face2Face software. (C,D) Teacher and learner have one-on-one time, and discuss mostly academic topics while the AR smartglasses provide rewards for mutual gaze. (E) Half the intervention sessions in this study were conducted within a general-education classroom. (F) The teacher succeeded in conducting the one-on-one session in the back of the classroom, while simultaneously managing the rest of the classroom. In this example other students were delivering presentations of their class projects. Consent was obtained to publish this photograph in a publicly accessible research journal.
Each intervention session was split into three segments, during which time the teacher was encouraged to engage in a natural conversation with the participant about relevant academic topics, for example a student's current project, homework assignments, or academic activities of the day. Each intervention lasted 10-min. The three phases were as follows:
1) “Pre-Face2Face,” in which the participant and facilitator conversed without the Empowered Brain for 1 min;
2) “Face2Face,” in which the participant and facilitator conversed while the participant wore Empowered Brain running the Face2Face module for 8 min; and
3) “Post-Face2Face,” in which the participant and facilitator conversed without the Empowered Brain for 1 min.
During the Face2Face phase (Phase 2), the participant would wear the Empowered Brain and would experience the feedback coaching of the Face2Face module as he conversed with the educator. During the conversation, the Face2Face module monitored the level of attention the participant was directing to the educator's face. The participant could gain game-like points and AR rewards when he looked toward the educator during the conversation. If the participant looked away from the educator, he would get visual and auditory guidance to help him redirect his attention back to the educator.
The Participant
The participant was a white Caucasian male aged 13 years and 11 months. He was diagnosed with ASD by his pediatrician at the age of 2. He receives special education services with an Individualized Education Program (IEP) at a mainstream public school in Massachusetts.
The participant previously had ASD-related interventions including applied behavioral analysis, occupational therapy, and speech and language therapy. He has previous experience with smartphone and tablet devices. He has no concurrent psychiatric disorders and was not receiving any psychotropic medication at the time of the study. Additionally, he has no history of epilepsy or seizures.
The Classroom Setting
This study was performed in a public middle school in Massachusetts, USA. The intervention was delivered twice-daily during school days. The first session was delivered by the student's general education teacher during the morning general education class. This classroom setting consisted of 2 educators, the general educator teacher and a paraprofessional, looking after a class of 23 students. The second daily session was delivered by the student's special education teacher during an afternoon special education class. The special education teacher was aided by a paraprofessional in looking after the 11 students in this class.
Consent and IRB Statement
The use of the Empowered Brain running on multiple head-worn computing devices by children and adults with ASD was approved by Asentral, Inc., Institutional Review Board, an affiliate of the Commonwealth of Massachusetts Department of Public Health. The study was performed in accordance with relevant guidelines and regulations, and in accordance with the Helsinki Declaration. Written informed consent was obtained from all parents/legal guardians of all minors involved in this study. Consent to conduct this research was also obtained from all educators involved in the study. Written informed consent was obtained from the participant's parent for the publication of their identifiable information.
Results
The baseline school week concluded with no concerns by the participant's teachers or parent (mother). All three school educators and parent completed the baseline SRS-2 at the end of the initial week.
The intervention was delivered to the participant on a total of 16 occasions over the 2-week intervention period, twice-daily during the 4 days of school in each week. School educators and the parent noted no usability or adverse effects that resulted in an intervention session being terminated early or being postponed. All three school educators and the participant's parent completed an SRS-2 at the end of Week 2 and Week 3, following eight total intervention sessions and 16 total intervention sessions, respectively. The results of the SRS-2 ratings are outlined in Tables 1–4 (and in graphic format in Figure 4).
Table 1. Special education teacher.
Table 2. General education teacher.
Table 3. Parent (mother).
Table 4. Paraprofessional educator.
Figure 4. Total and subscale Social Responsiveness Scale 2nd Edition (SRS-2) T scores for educators and parent at baseline and at the end of the intervention weeks.
Discussion
Emerging assistive technologies may help to augment the delivery of social communication interventions in educational settings. There is great demand for such technologies given the growing ASD student population and current lack of access to specialized resources. However, it is important to study several aspects of these technologies, including their usability, reliability, efficacy, but also the ability for users to use and benefit from the technologies in real-world settings. This is especially important when technologies are created for use in schools given the challenges in integrating assistive digital tools in the busy schedule and environment of a classroom.
The findings of our case study demonstrate preliminary evidence that this particular autism-focused intervention, the Empowered Brain, is associated with improvements in social communication. This improvement was observed through the use of the SRS-2, a gold standard and validated social communication scale. The student's parent (mother), special educator teacher, and general education teacher, reported an improvement in the SRS-2 global scale in addition to improvement in social communication, social cognition, social motivation, and restricted interests and repetitive behavior subscales. The paraprofessional, who only provided a single intervention, did not note any substantial change in the SRS-2 total score.
Our findings have a number of implications and limitations, adding to the current literature in several important ways. Firstly, it was evident that a wide variety of school educators were able to use this smartglasses intervention with the student, despite this being their first experience with smartglasses. The school educators were keen to support the student's social communication needs through technology. Secondly, the educators were able to use this technology in the school setting, and specifically within the same classroom that the child would otherwise be attending. Compared to a research setting, this dynamic classroom/school environment poses considerably more sensory, social, physical, and organizational challenges. Finally, the successful and timely completion of all 16 intervention sessions of Face2Face demonstrates that this is a practical and usable technology in this setting. At no time were any issues of usability or negative effects with the use of Face2Face noted, and lack of adverse effects have been previously reported in a larger population (Sahin et al., 2018).
The potential adoption of this technology may have broad implications on the educational system. Anecdotal feedback from educators involved in the study suggested that they found the 10 min intervention sessions to be an important time period for assessment and relationship building. This has implications for the larger context of education, as it provides the teacher with additional tools (data and trust/social capital) to increase personal agency and effectiveness. The findings may also indicate a role for this intervention to address the socio-communicative challenges that often interfere with student engagement and teacher effectiveness in schools. Classrooms endeavoring to deliver an inclusive learning experience must balance administering differentiated content-based instruction that engages the unique abilities and needs of a neurodiverse student population while simultaneously identifying individual-specific disruptions in the learning process and implementing targeted interventions informed by data derived from observation and assessment (Table 5). Smartglasses, worn by student, can potentially serve as a non-invasive mechanism (or platform) to administer and monitor differentiated learning content as well as user-specific targeted interventions. Smart glasses, worn by the teacher, can potentially help to facilitate the collection of observational data in real-time and provide immediate feedback for action as well as long term goal setting and strategy developed during the IEP process (this has not yet been tested). Smart glasses and augmented reality as platforms for differentiated instruction and targeted intervention in schools (as described above) can theoretically be applied to every student in the classroom (based on their specific needs) and can fundamentally change the way academic content and curriculum is delivered as well as impacting the way educators are trained and evaluated. Software development for the device that can effectively address many of the pain points and lingering challenges educators and parents face is necessary to fully realize the potential of the platform.
Table 5. Advantages for the educational system.
Consideration must be given to a number of limitations of the study, both in regards to methodology and the findings. Firstly, this report is a case study, and there are limitations to the generalizability of our findings to the broader ASD student population. We have also noted that we had a limited number of data points in our study. While research on smartglasses in ASD has been quite limited, cohort studies following multiple learners would help to improve our understanding of the efficacy of such interventions across the spectrum. Further testing of this technology requires continuous measures to supplement the pre- and post-measures that have been obtained in this study. Additionally, it is important to replicate these results in order to understand causality.
Additionally, we have designed this intervention for use in the educational system of the United States. There is considerable variance in the curriculum, use of assistive technology, and availability of special education resources in educational systems across the world. While some systems would share many similarities with the United States, there may be a need to make cultural and country-specific adjustments to the technology. The authors note that the base hardware of the Empowered Brain, Google Glass, was never available for purchase in many countries across the world. Importantly, there are a number of additional challenges to the adoption of this technology by the educational system, including educator attitudes and cost-implications (Table 6).
Table 6. Potential challenges for broader adoption.
As with any intervention, there is always a possibility of a placebo effect. The smartglasses provide a game-like experience that augments a person's normal perceptual experience. It is possible that such a novel experience is particularly prone to a placebo effect on the student, as well as the raters. On the other hand, people with ASD struggle with new experiences, and may also demonstrate extreme reactions to experiences that are difficult to their normal schedule, or that incorporate novel sensory stimuli (American Psychiatric Association, 2013). It was therefore reassuring that over 16 sessions, this digital perceptual experience did not result in these well-documented ASD-related adverse behaviors such as tantrums and/or meltdowns.
In future studies, we intend to broaden the scope of our target population to include students with other cognitive-based learning disabilities. While measuring the effectiveness of particular apps, we will also monitor the impact on overall quality of time-on-task for teachers and students. Social and behavioral issues often interfere with both a teacher's ability to convey content and skill, and the students' ability to receive and absorb that content and skill. If our intervention can increase attending and reduce behavioral distractions, it should also have a positive impact on the quantity and quality of time spent “on task” for both teachers and students. If this proves to be the case, we can begin to predict and plan for how much time “on task” will be required to move toward proficiency and mastery for each child in the classroom.
Availability of Data and Material
Collected data (SRS-2) is included in this paper.
Author Contributions
NS, JS, RL, RA-S, NK, and AV designed and undertook the intervention. The writing of this technology report was led by AV, and all authors contributed.
Funding
The authors have received generous funding support from the Game On For Autism Research (GoFAR) Foundation and Leslie S. T. Fang Foundation. Additionally, this work has been supported by the Office of the Assistant Secretary of Defense for Health Affairs, through the Autism Research Program under Award No. W81XWH 17-1-0449. Early work to transform smartglasses into biomedical sensors was supported in part by the United States Army Medical Research and Materiel Command under Contract No. W81XWH-14-C-0007 (awarded to TIAX, LLC). Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.
Conflict of Interest Statement
This report was supported by Brain Power, a neurotechnology company developing a range of artificially intelligent wearable technologies. Brain Power has engineering and technical partnerships with major technology companies and also receives funding support from federal and congressional sources. The authors all hold research or technical affiliations or employment with Brain Power. NS is the inventor of the Empowered Brain, and has multiple patents pending worldwide in relation to this intervention.
Acknowledgments
The authors thank the participants for their time and important contributions to our work. The authors thank Google, Inc., (Mountain View, CA, USA) for a generous grant and also the Glass team at X (formerly Google X) for technical guidance on Glass software development. The authors also thank Amazon and Amazon Web Services for support, guidance, and access to pre-release artificial intelligence tools. The authors are also grateful to Affectiva for their emotion artificial intelligence technology that has helped to enable this work.
Abbreviations
AI, Artificial Intelligence; AR, Augmented Reality; ASD, Autism Spectrum Disorder; IEP, Individualized Education Program; SRS-2, Social Responsiveness Scale 2.
References
Amalfitano, J. G., and Kalt, N. C. (1977). Effects of eye contact on the evaluation of job applicants. J. Employ. Couns. 14, 46–48. doi: 10.1002/j.2161-1920.1977.tb00637.x
American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). Washington, DC: American Psychiatric Association.
Ashman, R., Banks, K., Philip, R. C., Walley, R., and Stanfield, A. C. (2017). A pilot randomised controlled trial of a group based social skills intervention for adults with autism spectrum disorder. Res. Autism Spectr. Disord. 43, 67–75. doi: 10.1016/j.rasd.2017.08.001
Baio, J., Wiggins, L., Christensen, D. L., Maenner, M. J., Daniels, J., Warren, Z., et al. (2018). Prevalence of autism spectrum disorder among children aged 8 years – autism and developmental disabilities monitoring Network, 11 Sites, United States, 2014. MMWR Surveill. Summ. 67, 1–23. doi: 10.15585/mmwr.ss6706a1
Baron-Cohen, S. (1991). Do people with autism understand what causes emotion? Child Dev. 62, 385–395. doi: 10.2307/1131011
Baron-Cohen, S., Campbell, R., Karmiloff-Smith, A., Grant, J., and Walker, J. (1995). Are children with autism blind to the mentalistic significance of the eyes? Br. J. Dev. Psychol. 13, 379–398. doi: 10.1111/j.2044-835X.1995.tb00687.x
Barton, E. E., Pustejovsky, J. E., Maggin, D. M., and Reichow, B. (2017). Technology-aided instruction and intervention for students with ASD: a meta-analysis using novel methods of estimating effect sizes for single-case research. Remedial Spec. Educ. 38, 371–386. doi: 10.1177/0741932517729508
Bauminger, N., and Kasari, C. (2000). Loneliness and friendship in high-functioning children with autism. Child Dev. 71, 447–456. doi: 10.1111/1467-8624.00156
Bauminger-Zviely, N., Eden, S., Zancanaro, M., Weiss, P. L., and Gal, E. (2013). Increasing social engagement in children with high-functioning autism spectrum disorder using collaborative technologies in the school environment. Autism 17, 317–339. doi: 10.1177/1362361312472989
Bellini, S. (2006). The development of social anxiety in adolescents with autism spectrum disorders. Focus Autism Other Dev. Disabl. 21, 138–145. doi: 10.1177/10883576060210030201
Bennett, K. D., Ramasamy, R., and Honsberger, T. (2013). Further examination of covert audio coaching on improving employment skills among secondary students with autism. J. Behav. Educ. 22, 103–119. doi: 10.1007/s10864-013-9168-2
Borckardt, J. J., Nash, M. R., Murphy, M. D., Moore, M., Shaw, D., and O'neil, P. (2008). Clinical practice as natural laboratory for psychotherapy research: a guide to case-based time-series analysis. Am. Psychol. 63, 77-95. doi: 10.1037/0003-066X.63.2.77
Chevallier, C., Kohls, G., Troiani, V., Brodkin, E. S., and Schultz, R. T. (2012). The social motivation theory of autism. Trends Cogn. Sci. 16, 231–239. doi: 10.1016/j.tics.2012.02.007
Constantino, J. N., and Gruber, C. P. (2012). Social Responsiveness Scale (SRS): Western Psychological Services. Torrance, CA.
Corbett, B. A., Schupp, C. W., Simon, D., Ryan, N., and Mendoza, S. (2010). Elevated cortisol during play is associated with age and social engagement in children with autism. Mol. Autism 1:13. doi: 10.1186/2040-2392-1-13
Corbett, B. A., Swain, D. M., Coke, C., Simon, D., Newsom, C., Houchins-Juarez, N., et al. (2014). Improvement in social deficits in autism spectrum disorders using a theatre-based, peer-mediated intervention. Autism Res. 7, 4–16. doi: 10.1002/aur.1341
Corden, B., Chilvers, R., and Skuse, D. (2008). Avoidance of emotionally arousing stimuli predicts social-perceptual impairment in Asperger's syndrome. Neuropsychologia 46, 137–147. doi: 10.1016/j.neuropsychologia.2007.08.005
Corona, L. L., Christodulu, K. V., and Rinaldi, M. L. (2017). Investigation of school professionals' self-efficacy for working with students with ASD: impact of prior experience, knowledge, and training. J. Posit. Behav. Interv. 19, 90–101. doi: 10.1177/1098300716667604
Dawson, G., Rogers, S., Munson, J., Smith, M., Winter, J., Greenson, J., et al. (2010). Randomized, controlled trial of an intervention for toddlers with autism: the early start denver model. Pediatrics 125, e17–e23. doi: 10.1542/peds.2009-0958
Dickson, K. S., Suhrheinrich, J., Rieth, S. R., and Stahmer, A. C. (2017). Parent and teacher concordance of child outcomes for youth with autism spectrum disorder. J. Autism Dev. Disord. 48, 1423–1435. doi: 10.1007/s10803-017-3382-z.
Dingfelder, H. E., and Mandell, D. S. (2011). Bridging the research-to-practice gap in autism intervention: an application of diffusion of innovation theory. J. Autism Dev. Disord. 41, 597–609. doi: 10.1007/s10803-010-1081-0
Fast, Y. (2004). Employment for Individuals with Asperger Syndrome or Non-Verbal Learning Disability: Stories and Strategies. London, UK: Jessica Kingsley Publishers.
Grynszpan, O., Weiss, P. L., Perez-Diaz, F., and Gal, E. (2014). Innovative technology-based interventions for autism spectrum disorders: a meta-analysis. Autism 18, 346–361. doi: 10.1177/1362361313476767
Hendricks, D. (2010). Employment and adults with autism spectrum disorders: challenges and strategies for success. J. Vocat. Rehabil. 32, 125–134. doi: 10.3233/JVR-2010-0502
Hess, K. L., Morrier, M. J., Heflin, L. J., and Ivey, M. L. (2008). Autism treatment survey: services received by children with autism spectrum disorders in public school classrooms. J. Autism Dev. Disord. 38, 961–971. doi: 10.1007/s10803-007-0470-5
Holmes, J., and Fillary, R. (2010). Handling small talk at work: challenges for workers with intellectual disabilities. Int. J. Disabil. Dev. Educ. 47, 273–291. doi: 10.1080/713671114
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., and Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Except Child. 71, 165–179. doi: 10.1177/001440290507100203
Hurlbutt, K., and Chalmers, L. (2004). Employment and adults with Asperger syndrome. Focus Autism Other Dev. Disabl. 19, 215–222. doi: 10.1177/10883576040190040301
Jack, R. E., and Schyns, P. G. (2015). The human face as a dynamic tool for social communication. Curr. Biol. 25, R621–R634. doi: 10.1016/j.cub.2015.05.052
Jones, W., Carr, K., and Klin, A. (2008). Absence of preferential looking to the eyes of approaching adults predicts level of social disability in 2-year-old toddlers with autism spectrum disorder. Arch. Gen. Psychiatry 65, 946–954. doi: 10.1001/archpsyc.65.8.946
Kee, K. S., Green, M. F., Mintz, J., and Brekke, J. S. (2003). Is emotion processing a predictor of functional outcome in schizophrenia? Schizophr. Bull. 29:487. doi: 10.1093/oxfordjournals.schbul.a007021
Keshav, N. U., Salisbury, J. P., Vahabzadeh, A., and Sahin, N. T. (2017). Social communication coaching smartglasses: well tolerated in a diverse sample of children and adults with autism. JMIR Mhealth Uhealth 5:E140. doi: 10.2196/mhealth.8534
Kirchner, J. C., Hatri, A., Heekeren, H. R., and Dziobek, I. (2011). Autistic symptomatology, face processing abilities, and eye fixation patterns. J. Autism Dev. Disord. 41, 158–167. doi: 10.1007/s10803-010-1032-9
Klin, A., Jones, W., Schultz, R., Volkmar, F., and Cohen, D. (2002). Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Arch. Gen. Psychiatry 59, 809–816. doi: 10.1001/archpsyc.59.9.809
Kolbe, T., Partridge, M., and O'Reilly, F. (2012). Time and Learning in Schools: A National Profile. National Center on Time & Learning.
Kratochwill, T. R., and Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: randomization to the rescue. Psychol. Methods 15, 124–144. doi: 10.1037/a0017736
Lawton, K., and Kasari, C. (2012). Teacher-implemented joint attention intervention: pilot randomized controlled study for preschoolers with autism. J. Consult. Clin. Psychol. 80, 687–693. doi: 10.1037/a0028506
Lee, K., Eskritt, M., Symons, L. A., and Muir, D. (1998). Children's use of triadic eye gaze information for “mind reading”. Dev. Psychol. 34, 525–539. doi: 10.1037/0012-1649.34.3.525
Liu, R., Salisbury, J. P., Vahabzadeh, A., and Sahin, N. T. (2017). Feasibility of an autism-focused augmented reality smartglasses system for social communication and behavioral coaching. Front. pediatr. 5:145. doi: 10.3389/fped.2017.00145
Locke, J., Kasari, C., Rotheram-Fuller, E., Kretzmann, M., and Jacobs, J. (2013). Social network changes over the school year among elementary school-aged children with and without an autism spectrum disorder. Sch. Ment. Health 5, 38–47. doi: 10.1007/s12310-012-9092-y
Locke, J., Olsen, A., Wideman, R., Downey, M. M., Kretzmann, M., Kasari, C., et al. (2015). A tangled web: the challenges of implementing an evidence-based social engagement intervention for children with autism in urban public school settings. Behav. Ther. 46, 54–67. doi: 10.1016/j.beth.2014.05.001
Lopata, C., Volker, M. A., Putnam, S. K., Thomeer, M. L., and Nida, R. E. (2008). Effect of social familiarity on salivary cortisol and self-reports of social anxiety and stress in children with high functioning autism spectrum disorders. J. Autism Dev. Disord. 38, 1866–1877. doi: 10.1007/s10803-008-0575-5
McMillan, J. H. (2004). “Educational research,” in Fundamentals for the Consumer, 4th Edn, ed J. H. McMillan (Boston, MA: Pearson/A and B), 379.
McMurray, K., and Pierson, M. R. (2016). The importance of assistive technology in the classroom for males and females with autism spectrum disorder. J. Gend. Pow. 5, 59–78.
Moriuchi, J. M., Klin, A., and Jones, W. (2017). Mechanisms of diminished attention to eyes in autism. Am. J. Psychiatry 174, 26–35. doi: 10.1176/appi.ajp.2016.15091222
Nelson, A. T., Lopata, C., Volker, M. A., Thomeer, M. L., Toomey, J. A., and Dua, E. (2016). Exploratory factor analysis of SRS-2 teacher ratings for youth with ASD. J. Autism Dev. Disord. 46, 2905–2915. doi: 10.1007/s10803-016-2822-5
Ohl, A., Grice Sheff, M., Little, S., Nguyen, J., Paskor, K., and Zanjirian, A. (2000). Predictors of employment status among adults with autism spectrum disorder. Work 56, 345–355. doi: 10.3233/WOR-172492.
Pascalis, O., de Vivies, X. D., Anzures, G., Quinn, P. C., Slater, A. M., Tanaka, J. W., et al. (2011). Development of face processing. Wiley Interdiscip. Rev. Cogn. Sci. 2, 666–675. doi: 10.1002/wcs.146
Pavlova, M. A., Guerreschi, M., Tagliavento, L., Gitti, F., Sokolov, A. N., Fallgatter, A. J., et al. (2017). Social cognition in autism: face tuning. Sci. Rep. 7:2734. doi: 10.1038/s41598-017-02790-1
Robey, R. R., Schultz, M. C., Crawford, A. B., and Sinner, C. A. (1999). Single-subject clinical-outcome research: designs, data, effect sizes, and analyses. Aphasiology 13, 445–473. doi: 10.1080/026870399402028
Rogers, S. J. (2000). Interventions that facilitate socialization in children with autism. J. Autism Dev. Disord. 30, 399–409. doi: 10.1023/A:1005543321840
Ryan, A. M., and Patrick, H. (2001). The classroom social environment and changes in adolescents' motivation and engagement during middle school. Am. Educ. Res. J. 38, 437–460. doi: 10.3102/00028312038002437
Sahin, N. T., Keshav, N. U., Salisbury, J. P., and Vahabzadeh, A. (2018). Safety and Lack of negative effects of wearable augmented-reality social communication aid for children and adults with autism. J. Clin. Med. 7:e188. doi: 10.3390/jcm7080188
Shattuck, P. T., Narendorf, S. C., Cooper, B., Sterzing, P. R., Wagner, M., and Taylor, J. L. (2012). Postsecondary education and employment among youth with an autism spectrum disorder. Pediatr. Peds. 129, 1042–1049. doi: 10.1542/peds.2011-2864
Shindorf, Z. (2016). Exploring Barriers to the Generalization of Social Skills Interventions for Children Diagnosed with ASD: A Qualitative Analysis of 'Youth Engagement Through Intervention'. Graduate Student Theses, Dissertations, & Professional Papers. University of Montana.
Smith, J. D. (2012). Single-case experimental designs: a systematic review of published research and current standards. Psychol. Methods 17, 510–550. doi: 10.1037/a0029312
Snyder, T. D., de Brey, C., and Dillow, S. A. (2016). Digest of Education Statistics 2014, NCES 2016-006. National Center for Education Statistics.
Tanaka, J. W., and Sung, A. (2016). The “Eye Avoidance” hypothesis of autism face processing. J. Autism Dev. Disord. 46, 1538–1552. doi: 10.1007/s10803-013-1976-7
Tang, J., Falkmer, M., Horlin, C., Tan, T., Vaz, S., and Falkmer, T. (2015). Face recognition and visual search strategies in autism spectrum disorders: amending and extending a recent review by weigelt et al. PLoS ONE 10:e0134439. doi: 10.1371/journal.pone.0134439
Tottenham, N., Hertzig, M. E., Gillespie-Lynch, K., Gilhooly, T., Millner, A. J., and Casey, B. J. (2014). Elevated amygdala response to faces and gaze aversion in autism spectrum disorder. Soc. Cogn. Affect. Neurosci. 9, 106–117. doi: 10.1093/scan/nst050
Vahabzadeh, A., Keshav, N. U., Salisbury, J. P., and Sahin, N. T. (2018). Improvement of attention-deficit/hyperactivity disorder symptoms in school-aged children, adolescents, and young adults with autism via a digital smartglasses-based socioemotional coaching aid: short-term, uncontrolled pilot study. JMIR Ment. Health 5:e25. doi: 10.2196/mental.9631
Van Laarhoven, T., Winiarski, L., Blood, E., and Chan, J. M. (2012). Maintaining vocational skills of individuals with autism and developmental disabilities through video modeling. Educ. Training Autism Dev. Disabil. 47, 447–461. Available online at: https://www.jstor.org/stable/23879638.
von dem Hagen, E. A., and Bright, N. (2017). High autistic trait individuals do not modulate gaze behaviour in response to social presence but look away more when actively engaged in an interaction. Autism Res. 10, 359–368. doi: 10.1002/aur.1666
Wainer, A. L., and Ingersoll, B. R. (2011). The use of innovative computer technology for teaching social communication to individuals with autism spectrum disorders. Res. Autism Spectr. Disord. 5, 96–107. doi: 10.1016/j.rasd.2010.08.002
Walsh, E., Holloway, J., McCoy, A., and Lydon, H. (2016). Technology-aided interventions for employment skills in adults with autism spectrum disorder: a systematic review. Rev. J. Autism Dev. Disord. 4, 12–25. doi: 10.1007/s40489-016-0093-x
Watkins, L., O'Reilly, M., Ledbetter-Cho, K., Lang, R., Sigafoos, J., Kuhn, M., et al. (2017). A meta-analysis of school-based social interaction interventions for adolescents with autism spectrum disorder. Rev. J. Autism Dev. Disord. 4, 277–293. doi: 10.1007/s40489-017-0113-5
Werts, M. G., Wolery, M., Snyder, E. D., and Caldwell, N. K. (1996). Teachers' perceptions of the supports critical to the success of inclusion programs. J. Assoc. Pers. Sev. Handicaps 21, 9–21. doi: 10.1177/154079699602100104
White, S. E. (2014). Special education complaints filed by parents of students with autism spectrum disorders in the midwestern United States. Focus Autism Other Dev. Disabl. 29, 80–87. doi: 10.1177/1088357613478830
Keywords: autism, assistive technology, digital health, augmented reality, artificial intelligence, mental health, social communication, special education
Citation: Sahin NT, Abdus-Sabur R, Keshav NU, Liu R, Salisbury JP and Vahabzadeh A (2018) Case Study of a Digital Augmented Reality Intervention for Autism in School Classrooms: Associated With Improved Social Communication, Cognition, and Motivation via Educator and Parent Assessment. Front. Educ. 3:57. doi: 10.3389/feduc.2018.00057
Received: 03 April 2018; Accepted: 25 June 2018;
Published: 13 September 2018.
Edited by:
Tom Crick, Swansea University, United KingdomReviewed by:
Ning Gu, University of South Australia, AustraliaMats Granlund, Jönköping University, Sweden
Copyright © 2018 Sahin, Abdus-Sabur, Keshav, Liu, Salisbury and Vahabzadeh. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ned T. Sahin, sahin@post.harvard.edu