Evaluating the Usability of a Wearable Social Skills Training Technology for Children with Autism Spectrum Disorder

Kinsella, Ben G.; Chow, Stephanie; Kushki, Azadeh

doi:10.3389/frobt.2017.00031

ORIGINAL RESEARCH article

Front. Robot. AI, 12 July 2017

Sec. Biomedical Robotics

Volume 4 - 2017 | https://doi.org/10.3389/frobt.2017.00031

This article is part of the Research Topic Human-Centered Rehabilitation and Assistive Robotics View all 5 articles

Evaluating the Usability of a Wearable Social Skills Training Technology for Children with Autism Spectrum Disorder

$\r\nBen G. Kinsella$ Ben G. Kinsella¹

Stephanie Chow² $Azadeh Kushki,*\r\n$ Azadeh Kushki^1,2*

¹Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, ON, Canada
²Autism Research Centre, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada

Affecting 1 in 68, autism spectrum disorder (ASD) is a complex neurodevelopmental disorder characterized by social skill impairments. While prognosis can be significantly improved with intervention, few evidence-based interventions exist for social skill deficits in ASD. Existing interventions are resource-intensive, their outcomes vary widely for different individuals, and they often do not generalize to new contexts. Technology-aided intervention is a motivating, low-cost, and versatile approach for social skills training in ASD. Although early studies support the feasibility of technology-aided intervention, existing approaches have been criticized for teaching social skills through human-to-computer interaction, paradoxically leading to increased social isolation. To address this gap, we propose a system to help guide human-to-human interaction called Holli, a wearable technology to serve as a social skills coach for children with ASD. The Google Glass-based application listens to conversations and prompts the user with appropriate social responses. In this paper, we describe a usability study we conducted to determine the feasibility of using wearable technology to prompt children with ASD throughout social conversations. Fifteen children with ASD (mean age = 12.92 ± 2.33, verbal intelligent quotient = 103.3 ± 18.73) used the application while engaging in a restaurant-themed interaction with a research assistant. The application was evaluated on its effectiveness (i.e., how accurately the application responds), efficiency (i.e., how quickly the user and the application respond), and user satisfaction (based on a post-session questionnaire). All users were able to successfully complete the 10-turn exchange while using Holli. The results indicated the Holli accurately detected and recognized user utterance in real time. Participants reported positive experiences of using the application. To the best of our knowledge, this system is the first technology-aided intervention for ASD that employs human-to-human social coaching, and our results demonstrate the device is a viable medium for treatment delivery.

Introduction

Autism spectrum disorder (ASD) is a complex neuro-developmental disorder characterized by impairments in social communication and the presence of restricted and repetitive behaviors. Affecting 1 in 68 (Developmental, D. M. N. S. Y., & 2010 Principal Investigators, 2014), ASD is a life-long condition that profoundly impacts individuals, families, and society at large. A defining and debilitating feature of ASD is difficulties in interpreting social information and participating in social interactions. Social impairment in ASD has been associated with negative outcomes such as unemployment, low rates of independent living, and increased risk of psychiatric disorders (e.g., anxiety and depression) (Howlin et al., 2013).

Various behavioral interventions for social skills have been proposed, including video modeling (D’Ateno et al., 2003), social stories (Gray, 2010), social skills training groups (Cappadocia and Weiss, 2011), cognitive behavioral therapy (Bauminger, 2007), and school-based training (Bellini et al., 2007). Further research for these social intervention strategies is still necessary to ensure training promotes generalization and peer acceptance (Autism Ontario, 2011). Many of the above interventions are expensive and difficult to implement due to the need for human resources and training. This issue is exacerbated by the fact that effective learning and generalization of skills requires rehearsal and repetition over periods of weeks or months. The resource needs of these interventions, therefore, significantly limit accessibility to treatment, as evidenced by service wait times that range from months to years (Autism Ontario, 2011). Some of these barriers can be reduced through the use of technology for treatment delivery.

Technology is a motivating learning medium for children with ASD (Williams et al., 2002; Hourcade et al., 2011), and several studies have shown that children’s attention, communication, and social skills improve when computers or tablets are used (Bernard-Opitz et al., 2001; Gal et al., 2009; Burke et al., 2013; Murdock et al., 2013). Moreover, the observation that most children with ASD show an affinity to computers (Moore, 1998) and a preference for the use of touchscreens (Hourcade et al., 2013), supporting the potential of technology as an effective and cost-efficient teaching tool for social skills. Specifically, the shift toward multi-touch tablet and mobile-based interventions as instructional and assistive tools offers children with ASD a socially acceptable means to experience a level of safety and control that may not be attained when interacting with other individuals (Konstantinidis et al., 2009). This approach to teaching new skills permits the development of a broad range of accessible applications for skill learning, behavior management, and facilitation of social communication (Bernard-Opitz et al., 2001; Williams et al., 2002; Gal et al., 2009; Hourcade et al., 2011; Burke et al., 2013).

Accordingly, the idea of applying technology to assist and teach children with ASD has been around for over 40 years (Colby, 1973), and there have been many studies that demonstrate its effectiveness (Weiss et al., 2011; Hourcade et al., 2013; Ploog et al., 2013). Various types of technology applications have been developed for children with ASD, including interactive robots (Zheng et al., 2013; Bekele et al., 2014), virtual characters (Bosseler and Massaro, 2003; Tartaro and Cassell, 2006; Hopkins et al., 2011), video games (Gotsis et al., 2010), and virtual reality (Ke and Im, 2013).

Technology-aided intervention may be used to reduce some of the barriers limiting access to social skills intervention. In particular, technology-aided intervention can provide a highly motivating medium for rehearsal of skills in a safe, controlled, and self-paced manner (Parsons and Beardon, 2000), allow for treatment programs to be implemented with high precision and fidelity and less variability, and reduce the cost of intervention and other accessibility barriers (Ploog et al., 2013). Despite this promise, technology-aided intervention in ASD is an emerging field (Ploog et al., 2013). An important criticism of existing technologies is teaching skills learned from human–computer interaction, which may paradoxically increase social isolation and hinder skill generalizability. To address this gap, we propose a system to help guide human-to-human interaction called Holli, a wearable technology to serve as a social skills coach for children with ASD.

Holli is currently implemented on the Google Glass platform (Google Inc., 2016). Feasibility of the Google Glass in clinical applications has been demonstrated in several studies: as an assistive tool for doctors in surgery (Muensterer et al., 2014), as a job interview guide for individuals with ASD (Xu et al., 2015), and to support dietary management for individuals with diabetes (Wall et al., 2014).

In this paper, we describe a usability study we conducted to determine the feasibility of using the Google Glass to prompt children with ASD throughout social conversations. Usability describes the ease of use of a particular device and how well matched it is for a user’s specific needs. This study is the first step in our system’s development, ensuring it meets the needs and is an acceptable medium for technology-based social skills coaching for children with ASD.

Materials and Methods

System Development

Google Glass

The Google Glass is a head-mounted display designed in the shape of eyeglasses, with lightweight titanium frames, as seen below in Figure 1. It can be worn as a standalone frame or mounted on top of prescription glasses. The Glass was chosen as a platform for our application because it includes a microphone and display while allowing the user to look at his/her conversation partner. The Glass has a touchpad on the side of the device that is used for navigation through the device’s user interface. Other than the touchpad, the device can be controlled through the device’s microphone and speech recognition capabilities. It also has a front facing camera that has the ability to take photos and record 720p HD videos, and it uses a clear display over the user’s right eye to overlay images onto the user’s field of view. At the time of the study, the cost of Holli was $1,500. When using Holli, the battery can last several hours.

FIGURE 1

Figure 1. The Google Glass hardware.

Holli

Holli is a software application developed using the Glass Development Kit, an open source software package, and running directly on the Google Glass. User and interaction partner speech are captured through the on-board microphone, as seen in Figure 2. The Google speech recognition engine is used to translate the spoken word into text. We have developed a rule-based system that processes the input text and generates appropriate responses, which are then displayed to the user through the head-mounted display. When the user utters one of the responses, prompts disappears and Holli begins to listen for the next turn in conversation. Figures 3 and 4 are screenshots taken from Holli demonstrating an example of what the user sees in each phase. In this example, the screen says “Listening…” until the interaction partner says a greeting, such as “Welcome.” Holli then provides various greetings for the user to choose from, as seen in Figure 4. When Holli recognizes the user’s response, the prompt disappears, and Holli begins to listen for the next exchange in conversation. This process can be seen in Figure 5.

FIGURE 2

Figure 2. Example use of Holli in a conversation.

FIGURE 3

Figure 3. The Holli user interface while in listening mode.

FIGURE 4

Figure 4. Examples of Holli prompts.

FIGURE 5

Figure 5. Overview of the Holli system.

Participants

A convenience sample of 15 children (male = 10) with ASD was recruited for this study. This sample size is sufficient for revealing at least 90% of usability problems (Faulkner, 2003). Our inclusion criteria were a clinical diagnosis of ASD, age between 8 and 16 years, verbal intelligent quotient (IQ) greater than or equal to 80, and able to read without glasses.

Participants were recruited through the Autism Research Center at Holland Bloorview Kids Rehabilitation Hospital. All participants were diagnosed by an expert clinical team, using DSM-IV criteria supported by the Autism Diagnostic Observation Schedule (Lord et al., 2000). Intellectual functioning was assessed using the Wechsler Abbreviated Scale of Intelligence (I and II). ASD symptom severity and anxiety was measured using the Social Communication Questionnaire. Other information collected on the participants included age, gender, comorbidities, and other received interventions.

The Bloorview Research Institute research ethics board approved the study. Written consent was obtained from all participants who were deemed to have the capacity for consent. For all other participants, assent and written consent were obtained from the children and their legal guardians, respectively.

Outcome Measures

This study investigates the usability of our design, based on ISO 92411-11 guidelines on usability evaluation. This standard recommends the use of measures of effectiveness, efficiency, and user satisfaction (Abran et al., 2003).

Effectiveness

This measure evaluates the system’s ability to accurately detect and recognize the participants’ exchanges. To this end, we used two effectiveness measures namely, detection accuracy and recognition accuracy. Detection accuracy, as defined below in Eq. 1, measures if the device detected user utterances. Detection accuracy is determined for both the user and the interaction partner and is calculated as follows.

Recognition accuracy, as defined below in Eq. 2, measures how accurately the device recognized specific phrases. Recognition accuracy is only determined for the user because an inaccuracy while recognizing the interaction partner’s speech may cause the conversation to diverge from the planned study script.

Both accuracy measures were computed by reviewing video recordings of the session and automatically generated logs that document the system’s detected phrases and the corresponding responses.

Efficiency

This measure evaluates the system’s speed in interpreting and responding to speech, as well as the user response time to prompts. The user response time is defined as the time between a prompt being displayed and the beginning of the user’s next utterance. The system response time is defined as the time between the end of an utterance and the system’s recognition of a phrase. These measures are determined by cross-examining the videos at 12.5% playback speed to determine the beginning and end of actual utterances, and logs created by Holli recording the timing of phrase detection and prompt display.

User Satisfaction

This aspect of usability was evaluated using a post-session satisfaction questionnaire, shown in Appendix A. The questionnaire included a 5-point Likert scale to measure the participants’ level of agreement with 10 statements. The questionnaire also included open-ended questions to help guide a semi-structured interview.

Procedures

After providing consent, the participants filled out a demographics questionnaire and were given an overview of Holli. The participants were guided through a restaurant-themed practice conversation. The restaurant-themed conversation included 10 unique exchanges and 128 unique words. Each exchange consists of one phrase from each conversation partner. The restaurant theme was chosen to emphasize functional interactions. The participants were given the Glass to adjust until comfortable. The Holli app was then started. A research assistant played the role of a restaurant staff (interaction partner) and prompted the participant for their order. Holli then provides possible responses to the participant using the heads-up display. The conversation continued in this manner for 10 interactions. Only the participant wore the Google Glass equipped with Holli. The interaction partner had no supplementary device, with the intent that their voice would be picked up by the user’s Glass. After the conversation, participants filled out a satisfaction questionnaire and we conducted a semi-structured interview.

Analysis

Descriptive statistics were computed for measures of effectiveness and efficiency, and for each item on the user satisfaction questionnaire. In addition, quantitative content analysis was undertaken to quantify the results from the semi-structured interview and extract several common themes.

Results

Participants

The participants’ mean age was 12.92 ± 2.33 with a mean verbal IQ of 103.3 ± 18.73 and mean SCQ score of 18.1 ± 4.42. Comorbidities included attention-deficit hyperactivity disorder (ADHD), tic disorders, and learning disorders. Other interventions varied across participants, but common elements include applied behavior analysis, speech therapy, occupational therapy, physiotherapy, intensive behavioral intervention, relationship development intervention, and cognitive behavioral training. Participant characteristics are shown below in Table 1.

TABLE 1

Table 1. Participant characteristics.