A Feasibility and Acceptability Trial of Social Cognitive Therapy in Early Psychosis Delivered Through a Virtual World: The VEEP Study

Background: Addressing specific social cognitive difficulties is an important target in early psychosis and may help address poor functional outcomes. However, structured interventions using standard therapy settings including groups suffer from difficulties in recruitment and retention.

Aims: To address these issues, we aimed to modify an existing group social cognitive intervention entitled ‘Social Cognition and Interaction Training' (SCIT) to be delivered through a virtual world environment (Second Life ©).

Methods: A single arm nonrandomized proof-of-concept trial of SCIT-VR was conducted. Five groups of three to five individuals per group were recruited over 6 months. Eight sessions of SCIT-VR therapy were delivered through the virtual world platform Second Life© over a 5-week intervention window. Feasibility was examined using recruitment rates and retention. Acceptability was examined using qualitative methods. Secondary outcomes including social cognitive indices, functioning, and anxiety were measured pre- and postintervention.

Results: The SCIT-VR therapy delivered was feasible (36% consent rate and 73.3% intervention completion rate), acceptable (high overall postsession satisfaction scores) and safe (no serious adverse events), and had high levels of participant satisfaction. Users found the environment immersive. Prepost changes were found in emotion recognition scores and levels of anxiety. There were no signs of clinical deterioration on any of the secondary measures.

Conclusion: This proof-of-concept pilot trial suggested that delivering SCIT-VR through a virtual world is feasible and acceptable. There were some changes in prepost outcome measures that suggest the intervention has face validity. There is sufficient evidence to support a larger powered randomized controlled trial.

Clinical Trial Registration: ISRCTN, identifier 41443166

Introduction

Novel treatments targeted at functional outcome in early psychosis are important as functional recovery is less common than symptomatic recovery (1), and the two are not always intrinsically linked (2). Continued functional impairment has significant impact both personally and economically and is often rated as the most important treatment goal in people experiencing their first episode of psychosis (3). One treatment approach targeting functional outcomes broadly is social cognition remediation therapy. Social cognition, including the domains of emotion recognition, theory of mind, and attributional bias, is often poor in both established psychosis (4) and first episode psychosis (5), is strongly related to social functioning (6), and contributes unique variance in predicting functional outcome (7, 8). Although varied in approach, social cognitive remediation programs all appear to improve social cognition measures (9). Global approaches that deliver both social cognition skills training and real-world application of these skills also appear to improve functional outcome at least in established psychosis (9). In a previous study, we adapted one of these established global type social cognition programs, Social Cognition and Interaction Training (10), in an Australian sample of first episode psychosis patients (11). We found in a small pilot study that this group intervention was acceptable to young people, but feasibility was low as consent rate and maintaining attendance in the group was relatively poor (11). For some young people, the thought of attending traditional individual or group therapy provoked considerable anxiety (11). In this sense, we were aware that our intervention was not reaching a wider group who might benefit from social cognition training, especially as poor social cognition is associated with anxiety and negative symptoms (12).

Over the last few decades, technological approaches to delivering healthcare have developed rapidly. This is also the case in mental health problems where the use of apps and social media sites has been used to encourage engagement and widen the reach of interventions to other populations (13). Although emerging, the use of technology to deliver interventions has received relatively little attention in the psychosis field, perhaps due to concerns regarding young people with psychosis not using technology or that technology may be unhelpfully incorporated into their symptoms (14). However, there is no consistent evidence to suggest either of these concerns are the case (13, 15).

Virtual Reality (VR) interventions are showing promise in engaging and helping young people with psychosis often by providing safe exposure to challenging environments. Virtual world platforms are being used to deliver physical and psychological treatments to people with a range of health problems, including obesity, autism, intellectual disabilities, and diabetes (16–19). A virtual world is an online shared community environment where members can interact in a custom-built, simulated world. The community interacts in the simulated world using either text-based, 2-D and/or 3-D graphical models called avatars (20). Although under-researched, virtual world platforms have the potential to reduce communication barriers and improve access to support and treatment in people with mental health disorders (21).

One of most widely used virtual world environments is the online and freely available social world Second Life©. Second Life© has been used as a virtual world platform in healthcare education, delivery, and engagement and has considerable potential in delivering psychological therapies (21, 22). For example, Second Life© has been used to treat social anxiety and has been found to be feasible and acceptable with effects sizes potentially comparable to face-to-face Cognitive Behavior Therapy (CBT) for Seasonal Affective Disorder (23) and in treating social cognition problems in autism (24). However, we know of no other study that has used Second Life© in the treatment of psychosis. We adapted a traditionally face-to-face delivered social cognition and interactional training (SCIT) (25, 26) to a virtual world in Second Life© (SCIT-VR) using a codesign process involving young people, clinicians and web designers. This work is described in detail elsewhere (27). The title of this trial is “virtual reality as a method of delivering social cognitive therapy in early psychosis (VEEP).” Our aim was to test the feasibility and acceptability of the SCIT-VR intervention approach in a group of young people who were recovering from a first episode of psychosis.

Methods

Study Design

Single arm, nonrandomized, proof-of-concept study.

Development of SCIT-VR

We undertook a modified codesign process with 20 young service users in total, two carers, a virtual world designer, and the study team. The co-design process is described in detail elsewhere [(27); Figure 1 displays screenshots of the environment].

FIGURE 1

Figure 1 Screenshots of the Second Life© environment adapted from (20).

SCIT-VR Structure

SCIT-VR consisted of 10 sessions (two individual and eight group sessions). The original SCIT (25) was delivered in 20 sessions but our adapted face-to-face version in early psychosis with young people living in Australia was delivered in 10 sessions (11). In the SCIT-VR version described in the current study, the group experience sessions at the end of the intervention were truncated based on our codesign feedback. We included two initial 30 minute sessions where the researcher helped the participant to set up and familiarize themselves with the technology in order to take part in the virtual group sessions. This included providing a headset, helping the participant through the login procedure, setting up their avatar, and helping to familiarize themselves with the Second Life© environment. The SCIT-VR content was then delivered over eight one-hour sessions. The basic structure of the intervention was the same as the original SCIT (25). The first three sessions focused primarily on emotion recognition, the next three sessions focused on attribution bias and paranoia as an emotion, and the last two sessions on “skills acquisition” using a CBT framework to discuss examples of social difficulties faced by the participant (see Figure 2). The intervention was manualized and delivered by a lead therapist and a cotherapist to groups of five participants.

FIGURE 2

Figure 2 SCIT-VR therapy structure.

Recruitment and Eligibility Criteria

Participants were recruited primarily from one National Health Service (NHS) early intervention in psychosis services in Coventry and Warwickshire, UK. These services provide a three year period of care to around 120 individuals experiencing their first episode of psychosis. Psychosis-specific interventions include antipsychotic medication, CBT, family interventions, and vocational interventions, plus individual case management. All young people aged between 18 and 45 years registered with the early intervention service and not in an acute phase of psychosis (i.e. not in hospital/in contact with the crisis team) were eligible. Exclusion criteria included a confirmed diagnosis of a learning disability or a neurological disorder.

Measures

Feasibility and Acceptability

Feasibility was assessed using consent rates in those eligible, levels of session completion and follow-up rates (28). Study completers were classified as individuals who attended at least six of the eight sessions (the two set-up sessions and at least four sessions). Acceptability was assessed immediately following each session by asking participants to complete a five-item feedback questionnaire completed online through the dedicated study website. Acceptability was also assessed using a semi-structured interview with participants post-intervention. We also completed a focus group with clinicians whose patients were involved in the study to access clinicians' views to real world applicability. These findings will be discussed in a future paper. Interview data was thematically organized.

Outcome Measures

Social Cognition

1) Social Cognition Screening Questionnaire (SCSQ)—comprises of 10 short vignettes, which presents an interaction between a fictional character and the participant. It has five subscales: verbal memory, schematic inference, theory of mind, metacognition, and hostility bias (29). 2) Bell Lysaker Emotion Recognition Task (BLERT)—this is a test of one's ability to correctly identify seven emotional states using facial emotion recognition (and body cues) endorsed by the preliminary reports of the SCOPE study (30). 3) Hinting task (31)—a theory of mind task where participants are required to identify verbal hints in a conversation. The social cognition measures were informed from the results of the ongoing Social Cognition Psychometric Evaluation (SCOPE) study (30, 32). 4) Cognitive Style Questionnaire-Short Form (CSQ-SF; (33)—focuses on 24 hypothetical positive and negative events relating to successes and failures in academic achievement, employment, and interpersonal relationships. This questionnaire assesses how an individual attributes the outcome of events.

Social Function

Personal and Social Performance Scale [PSP; (34, 35)]—clinician reported measure to assess the severity of social functioning in those with acute symptoms of schizophrenia.

Quality of Life

EuroQual 5-D (36) scale—a widely used scale assessing quality of life through five items and a visual analog health “thermometer” rating.

Behavioral Change Intention

1) Theoretical Domains Framework—Domain Four (TDF—D4)—used to develop questionnaires including 12 theoretical domains of potential behavioral determinants. Domain four is called Beliefs about capabilities and measures participants' self-efficacy, self-confidence, and perceived competence (37).

2) Theoretical Domains Framework—Domains Eight and Nine—these correspond to Intentions and Goals in the Framework. Intentions involves measuring participants' decision to undertake a behavior or resolution in a particular way. Goals involves assessing participants' goal priorities, targets, and their intention to implement such goals into their life (37).

Presence in the Virtual World

Presence (a VR concept employed to test the extent to which the participant was “immersed” in the virtual world) was tested using the presence questionnaire (38).

Psychopathology

Brief Psychiatric Rating Scale (BPRS) (39)—assesses participants' psychopathology and symptom severity. Researchers were trained on administering this scale to gold standard ratings.

Neurocognition

1) National Adult Reading Test (NART)—a commonly used method in clinical settings for estimating premorbid intelligence levels of English-speaking individuals (40). 2) Trail Making Test—a widely used neuropsychological test of visual attention and task switching (41).

Adverse Events

These were collected in line with the University of Warwick Clinical Trials operating procedure.

Procedure

Eligible individuals were identified by the treating team. Potential participants were approached for consent by the research assistant. Consented participants completed the baseline assessment over a maximum two weeks. When at least three but preferably five participants were consented and baseline assessment completed, therapy sessions commenced. The time between recruitment of the first and last participant was short (no longer than six weeks). The two initial individual sessions were arranged before the group sessions were scheduled at the convenience of the participants. To encourage attendance, participants were contacted by text prior to each session. Participants were aware that the secondary therapist was available via a study phone for an hour prior and after each session both for safety reasons and to help with technology queries.

The intervention was delivered to groups of three to five participants. The primary and secondary group facilitators were psychology trainees educated at masters level/above. The intervention was manualized. A number of PowerPoint presentations and videos were retained from the original SCIT intervention and delivered through Second Life© as part of the sessions. Immediately following each session, participants were taken to a ‘virtual library' where they were invited to complete a session feedback sheet. Feedback was not available to the therapists until after the end of the intervention period. At the end of the 10 therapy sessions, participants completed the post-intervention assessments. They were also invited to complete a 30-minute interview with a research assistant to discuss views on the technology, the virtual world and the intervention itself. All interviews were topic-guide driven, audio-recorded and transcribed. Ethical approval for the study was obtained from the West Midlands—Solihull Research Ethics Committee.

Data Analysis

For the secondary outcome measures, pre- and post-test scores were compared using basic t tests. Demographic information on the participants was also collected and presented. Qualitative interviews were conducted post intervention, using a semi structured schedule. Interviews were conducted until data saturation was achieved (N = 15). A thematic analysis (42) method was used to analyze the data. Analyses were conducted using the software NVivo.

Results

Sample Characteristics and Feasibility

Descriptive data for the participants who consented to take part in the study (N = 19) are displayed in Table 1. More males than females were recruited and the age was relatively young (mean 25.6 years). IQ estimates were slightly above normal and the level of psychopathology was relatively high. Of all the young adults with First Episode Psychosis (FEP) in the service who were deemed eligible by the care coordinators and approached (53 over a 6-month period), 19 participants (36%) agreed to take part. Four of those participants who consented (21%) then withdrew from the study before completing outcome measures, leaving outcome data for 15 individuals. Reasons for withdrawal from the study were: unable to commit the time (n = 2); experiencing a recent bereavement (n = 1); not able to contact (n = 1). If participants attended four or more of the total eight sessions, they were considered a completer. Eleven of the 15 participants were deemed completers (73.3%). The mean (SD) number of VEEP sessions attended by participants was 4.94 (2.64), which was a mean attendance rate of 61.8% of the total eight available sessions.

TABLE 1

Table 1 Participant demographics, psychopathology and neurocognitive measures at baseline.

Acceptability

Feedback Questionnaires

Participants gave ratings out of five for each of the five items assessing acceptability on a feedback questionnaire completed after every session. Acceptability was rated >3 out of five on every item. Descriptive data for ratings of each item of the questionnaire are presented in Table 2, while total ratings (0–25) for each intervention session are presented in Table 3.

TABLE 2

Table 2 Acceptability ratings (0–5) for each item of the feedback questionnaire.

TABLE 3

Table 3 Total acceptability ratings (0–25) for each session of VEEP.

Qualitative Interviews

Categories and illustrative quotes from the individual interviews are represented in Table 4. These were related to investigator's concerns on safety and immersion/realism in the environment. Participants did not have specific concerns regarding safety, liked the anonymity, and found the environment relatively immersive. There were some issues with the technology reported by the participants.

TABLE 4

Table 4 Feedback and illustrative quotes.

Secondary Outcome Measures

Treatment outcome was defined as change in social cognition, social function, quality of life, and behavioral change intention. Paired t-tests showed that there was a significant increase in emotion recognition (BLERT) scores from pre- to post-intervention, with a medium effect size (Hedges' g), t(14) = 3.21, p = .006, g = 0.58, (see Figure 3). There was also a significant decrease in the anxiety/depression subscale of the EuroQual-5D, indicating an improvement with a medium effect size, t(14) = 2.43, p = .029, g = 0.41 (see Figure 4) (43, 44). There were no other significant differences from pre- to post-treatment on any of the other measures (p > .1). See Table 5 for pre- and post-intervention analyses. Presence was measured in the last nine consecutive participants (due to an omission in the interview schedule). The results showed that the participants had a reasonable degree of presence in the virtual world (average score on the presence questionnaire (38) 154.2, SD 18.8 see Supplementary Figure 1).

FIGURE 3

Figure 3 Bar chart to show means and 95% confidence intervals of BLERT scores from pre- to post-intervention.

FIGURE 4

Figure 4 Bar chart to show means and 95% confidence intervals of EuroQual-5D-Anxiety/depression subscale ratings from pre- to post-intervention.

TABLE 5

Table 5 Pre- and post-intervention means and SDs for secondary outcomes.

Discussion

We conducted a proof-of-concept nonrandomized study of an established group social cognition intervention (SCIT) adapted for delivery through a virtual world (Second Life©), SCIT-VR. We found that the intervention was generally feasible as the intervention completion rate was reasonable, and the delivery of the intervention by relatively inexperienced clinicians in an NHS setting was possible. Our completion rate is similar to that of the trial conducted by Bartholomeusz et al. (11) where 75% of participants completed the intervention. The intervention was mostly considered acceptable as measured by attendance rate, post-session feedback questionnaires and qualitative interviews. There were no concerns about safety either from adverse events or from individual participant interviews. There was a suggestion that the intervention had face validity as pre-post changes were observed on measures of social cognition and anxiety. There was no change in participants' intention to change or self-efficacy scores. The intervention was not primarily aimed at improving intention to engage in further therapy or social activities but we anticipated this might be a beneficial effect of the approach. Further studies might want to consider the use of virtual worlds as interventions aimed more directly at engagement and self-efficacy as part of a pathway to accessing more structured and face-to-face interventions.

The small scale pilot face-to-face version of this study was conducted by Bartholomeusz et al. (11). The average effect size score in their trial was d = 0.29, compared to our g = 0.22. Similarly this trial also found significant improvements on emotion recognition. This is not completely unexpected as we believe the virtual world is a mechanism to provide the SCIT intervention to those who might find it difficult to attend face-to-face therapy. The findings in the current study across a number of measures indicate that the virtual world intervention we have developed has the potential to improve social cognitive domains in those people with early psychosis. However, this is a small pilot study with a small sample size. Therefore, a large randomized controlled trial is required to determine whether providing SCIT in a virtual world can lead to significant improvements in this population.

This is the first study to our knowledge using a virtual world to deliver a structured group therapy in early psychosis. It was relatively ambitious, as a number of social cognition skills would appear to rely on face-to-face social contact to achieve mastery. However, there have been previous studies that have used Second Life© successfully with the aim of improving social cognition skills in those with high functioning autism (24). We believe that the didactic teaching approach that was delivered in Second Life© was possibly more successful than more group-focused sessions as measured by higher levels of feedback on the individual sessions that were more didactic in nature. The sessions with more specific teaching content were rated more highly by the participants than the sessions that involved group work. No participants highlighted that they enjoyed the group or specifically felt part of the group. Future studies using this approach may want to consider whether the intervention needs to be delivered in a group setting. Others, for example, have successfully used an individual approach when using virtual worlds to treat social anxiety (23).

Participants reported a high degree of presence in the virtual environment, which was surprising as the interaction with Second Life© is purely through a computer monitor and headset and not using a head mounted display or more traditional immersive technology and the interview themes also highlighted that people felt high levels of presence in the virtual world. However, the presence questionnaire was completed by a minority of participants. Previous research has attempted to assess the sense of presence using Second Life©; for example, nursing students reported relatively high degree of presence in Second Life©, which was related to attitude towards using the environment (45). We have considered whether a more immersive environment such as traditional VR or using 360 videos might lead to a greater acquisition of social cognition skills as they more approximate real-life situations.

These findings support the growing number of VR interventions for people who experience psychosis. According to a recent literature review, there are a distinct lack of studies examining the potential impact of VR interventions for early psychosis (46) Therefore, our trial is timely. The key advantage of VR is that real-time behavior can be observed when interacting in the VR world. Furthermore, the environment can be modified and controlled to elicit certain responses (47).

There are a number of limitations of the study. Firstly, there was no control group and therefore we cannot be sure that any improvements are due to the intervention. There were challenges in recruiting patients, which impacted on the feasibility of the study, with only 36% of eligible patients consenting to take part in the intervention. This differs from previous research, which showed that two thirds of patients with psychosis were willing to be contacted about taking part in research (48). However consent rates for those eligible participants diagnosed with psychosis differ between trials. For example, while Wood et al. (49) reported a 67% consent rate (30/45), Kanniston et al. (50) reported a 33% consent rate (1139/3417), and Schrank et al. (51) reported a mean consent rate of 40.0%. One possible reason for our recruitment rate is that some potential participants may not have had access to the internet or may have had limited digital skills, reflecting an ongoing concern around digital exclusion for some people. This issue is a concern more generally in digital mental health studies and requires further consideration in finding ways to reduce digital inequalities (52). More frequent contact between clinicians and researchers has been shown to improve recruitment. Therefore this should be reviewed in future research (53).

Whilst we had a reasonable retention rate, we had problems keeping people engaged in the study. There were some problems with the technology during some of the sessions which required the secondary therapist to rectify; on two occasions this was not possible. This was highlighted in the qualitative interviews. One of the limitations of using Second Life © was that it was not functional on tablets or hand-held devices and it could not be streamed through a gaming console. A number of participants did not own a laptop or desktop computer. We were able to loan a computer to participants who did not have access to one, but for those who do not own or have access to a computer participation in an intervention delivered on a platform such as Second Life© is compromised.

Given the results of this small proof-of-concept study, there is sufficient evidence to consider a full-scale efficacy trial in this population. Challenges in designing such a trial are whether to consider adjunctive experiences that might be more immersive such as combining other VR approaches. Based on the feedback, we would also need to consider whether all parts of the intervention need to be delivered in a group setting and how we might be able to overcome technical problems and access to those without a computer. The delivery of evidence based psychological therapies to this group continues to prove challenging, and the use of technologies such as Second Life© shows promise in widening access and engagement for this hard to reach group.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

This study was carried out in accordance with the recommendations of the 'West Midlands – Solihull Research Ethics Committee' with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the 'West Midlands – Solihull Research Ethics Committee'.

Author Contributions

AT, SB, MB, IV, and DT wrote the grant application for this feasibility and pilot trial. AT, AR, DT, and FE were responsible for coordinating the design of the virtual world. AT, AR, and FE conducted the co-design workshops. AT, FE, FL, and AR undertook the trial. AT drafted the manuscript. All investigators have been involved in revising the report, and all authors have seen and approved the final version.

Funding

The VEEP study was sponsored by the University of Warwick and supported by the PsyIMPACT programme of MQ: MQ16PI100011. Transforming mental health (Register charity in England/Wales: 113991 & Scotland: SC046075).

Conflict of Interest

SB is a director of a not-for-profit company aimed at commercialising mental health apps.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We are indebted to all young service users and students who participated in the codesign and testing of the virtual world. We are grateful to Professor David Penn and Dr David Roberts for their guidance and assistance and generous use of the original SCIT intervention resources. Thanks to Robin and Elli Winter at www.warmwinterarts.com for their expertise in developing, programming, and constantly improving the design of a beautiful virtual world that became the VEEP island. The VEEP study was sponsored by the University of Warwick and supported by the PsyIMPACT programme of MQ: Transforming mental health (Register charity in England/Wales: 113991 & Scotland: SC046075). The views expressed are those of the authors and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2020.00219/full#supplementary-material

References

1. Henry LP, Amminger GP, Harris MG, Yuen HP, Harrigan SM, Prosser AL, et al. The EPPIC follow-up study of first-episode psychosis: longer-term clinical and functional outcome 7 years after index admission. J Clin Psychiatry (2010) 71(6):716–28. doi: 10.4088/JCP.08m04846yel

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Alvarez-Jimenez M, Gleeson JF, Henry LP, Harrigan SM, Harris MG, Killackey E, et al. Road to full recovery: longitudinal relationship between symptomatic remission and psychosocial recovery in first-episode psychosis over 7.5 years. Psychol Med (2012) 42(3):595–606. doi: 10.1017/S0033291711001504

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Iyer SN, Mangala R, Anitha J, Thara R, Malla AK. An examination of patient-identified goals for treatment in a first-episode programme in Chennai, India. Early Interv Psychiatry (2011) 5(4):360–5. doi: 10.1111/j.1751-7893.2011.00289.x

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Penn DL, Sanna LJ, Roberts DL. Social cognition in schizophrenia: an overview. Schizophr Bull (2008) 34(3):408–11. doi: 10.1093/schbul/sbn014

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Thompson A, Papas A, Bartholomeusz C, Allott K, Amminger GP, Nelson B, et al. Social cognition in clinical “at risk” for psychosis and first episode psychosis populations. Schizophr Res (2012) 141(2-3):204–9. doi: 10.1016/j.schres.2012.08.007

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Couture SM, Penn DL, Roberts DL. The functional significance of social cognition in schizophrenia: a review. Schizophr Bull (2006) 32(Suppl 1):S44–63. doi: 10.1093/schbul/sbl029

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Pinkham AE, Penn DL. Neurocognitive and social cognitive predictors of interpersonal skill in schizophrenia. Psychiatry Res (2006) 143(2-3):167–78. doi: 10.1016/j.psychres.2005.09.005

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Hoe M, Nakagami E, Green MF, Brekke JS. The causal relationships between neurocognition, social cognition and functional outcome over time in schizophrenia: a latent difference score approach. Psychol Med (2012), 42(11):2287–99. doi: 10.1017/S0033291712000578

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Kurtz MM, Richardson CL. Social Cognitive Training for Schizophrenia: A Meta-Analytic Investigation of Controlled Research. Schizophr Bull (2012) 38(5):1092–104. doi: 10.1093/schbul/sbr036

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Penn D, Roberts DL, Munt ED, Silverstein E, Jones N, Sheitman B. A pilot study of social cognition and interaction training (SCIT) for schizophrenia. Schizophr Res (2005) 80(2-3):357–9. doi: 10.1016/j.schres.2005.07.011

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Bartholomeusz CF, Allott K, Killackey E, Liu P, Wood SJ, Thompson A. Social cognition training as an intervention for improving functional outcome in first-episode psychosis: a feasibility study. Early Interv Psychiatry (2013) 7(4):421–6. doi: 10.1111/eip.12036

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Lincoln TM, Mehl S, Kesting M-L, Rief W. Negative Symptoms and Social Cognition: Identifying Targets for Psychological Interventions. Schizophr Bull (2011) 37(suppl_2):S23–32. doi: 10.1093/schbul/sbr066

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Bucci S, Barrowclough C, Ainsworth J, Machin M, Morris R, Berry K, et al. Actissist: Proof-of-Concept Trial of a Theory-Driven Digital Intervention for Psychosis. Schizophr Bull (2018) 44(5):1070–80. doi: 10.1093/schbul/sby032

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile Phone Ownership and Endorsement of “mHealth” Among People With Psychosis: A Meta-analysis of Cross-sectional Studies. Schizophr Bull (2016) 42(2):448–55. doi: 10.1093/schbul/sbv132

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Thompson A, Gleeson J, Alvarez-Jimenez M. Should we be using digital technologies in the treatment of psychotic disorders? Aust N Z J Psychiatry (2018) 52(3):225–6. doi: 10.1177/0004867418757920

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Watson AJ, Grant RW, Bello H, Hoch DB. Brave new worlds: how virtual environments can augment traditional care in the management of diabetes. J Diabetes Sci Technol (2008) 2(4):697–702. doi: 10.1177/193229680800200422

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Hall V, Conboy-Hill S, Taylor D. Using Virtual Reality to Provide Health Care Information to People With Intellectual Disabilities: Acceptability, Usability, and Potential Utility. J Med Internet Res (2011) 13(4):e91. doi: 10.2196/jmir.1917

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Fengfeng K, Im T. Virtual-Reality-Based Social Interaction Training for Children with High-Functioning Autism. J Ed Res (2013) 106(6):441–61. doi: 10.1080/00220671.2013.832999

CrossRef Full Text | Google Scholar

19. Taylor M, Taylor D, Kulendran M, Gately P, Darzi A. Virtual worlds as a tool to facilitate weight management for young people. J Virtual Worlds Res (2013) 6(1):7026. doi: 10.4101/jvwr.v6i1.7026

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Bartle RA. Designing virtual worlds. London: New Riders Publishing (2003).

Google Scholar

21. Rehm IC, Foenander E, Wallace K, Abbott JM, Kyrios M, Thomas N. What Role Can Avatars Play in e-Mental Health Interventions? Exploring New Models of Client-Therapist Interaction. Front Psychiatry (2016) 7:186. doi: 10.3389/fpsyt.2016.00186

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Gorini A, Gaggioli A, Vigna C, Riva G. A second life for eHealth: prospects for the use of 3-D virtual worlds in clinical psychology. J Med Internet Res (2008) 10(3):e21. doi: 10.2196/jmir.1029

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Yuen EK, Herbert JD, Forman EM, Goetter EM, Comer R, Bradley J-C. Treatment of Social Anxiety Disorder Using Online Virtual Environments in Second Life. Behav Ther (2013) 44(1):51–61. doi: 10.1016/j.beth.2012.06.001

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB. Virtual Reality Social Cognition Training for Young Adults with High-Functioning Autism. J Autism Dev Disord (2012) 43(1):34–44. doi: 10.1007/s10803-012-1544-6

CrossRef Full Text | Google Scholar

25. Penn DL, Roberts DL, Combs D, Sterne A. Best practices: The development of the Social Cognition and Interaction Training program for schizophrenia spectrum disorders. Psychiatr Serv (2007) 58(4):449–51. doi: 10.1176/ps.2007.58.4.449

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Roberts DL, Penn DL. Social cognition and interaction training (SCIT) for outpatients with schizophrenia: A preliminary study. Psychiatry Res (2009) 166(2-3):141–7. doi: 10.1016/j.psychres.2008.02.007

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Reaple A, Elahi F, Bucci S, Birchwood M, Vlaeve I, Taylor D, et al. Co-designing a virtual world with young people to deliver social cogntion therapy in early psychosis. Early Interv Psychiatry (2020) 14(1):37–43. doi: 10.1111/eip.12804

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Eldridge SM. Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PloS One (2016) 11(3):e0150205. doi: 10.1371/journal.pone.0150205

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Roberts D, Fiszdon J, Tek C. Initial validity of the Social Cognition Screening Questionnaire (SCSQ). Schizophr Bull (2011) 31(Suppl 1):280. doi: 10.1111/pcn.12181

CrossRef Full Text | Google Scholar

30. Pinkham AE, Penn DL, Green MF, Harvey PD. Social Cognition Psychometric Evaluation: Results of the Initial Psychometric Study. Schizophr Bull (2016) 42(2):494–504. doi: 10.1093/schbul/sbv056

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Corcoran R. Inductive reasoning and the understanding of intention in schizophrenia. Cognit Neuropsychiatry (2003) 8(3):223– 235. doi: 10.1080/13546800244000319

CrossRef Full Text | Google Scholar

32. Pinkham AE, Penn DL, Green MF, Buck B, Healey K, Harvey PD. The social cognition psychometric evaluation study: results of the expert survey and RAND panel. Schizophr Bull (2014) 40(4):813–23. doi: 10.1093/schbul/sbt081

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Meins E, McCarthy-Jones S, Fernyhough C, Lewis G, Bentall RP, Alloy LB. Assessing negative cognitive style: Development and validation of a Short-Form version of the Cognitive Style Questionnaire. Pers Individ Dif (2012) 52(5):581–5. doi: 10.1016/j.paid.2011.11.026

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Morosini PL, Magliano L, Brambilla L, Ugolini S, Pioli R. Development, reliability and acceptability of a new version of the DSM-IV Social and Occupational Functioning Assessment Scale (SOFAS) to assess routine social funtioning. Acta Psych Scand (2000) 101(4):323–9. doi: 10.1034/j.1600-0447.2000.101004323.x

CrossRef Full Text | Google Scholar

35. Patrick DL, Burs T, Morosini P, Rothman M, Gagnon DD, Wild D, et al. Reliability, validity and ability to detect change of the clinician-rated Personal and Social Performance scale in patients with acute symptoms of schizophrenia. Curr Med Res Opin (2009) 25(2):325–38. doi: 10.1185/03007990802611919

PubMed Abstract | CrossRef Full Text | Google Scholar

36. EuroQol G. EuroQol - a new facility for the measurement of health-related quality of life. Health Policy (1990) 16(3):199–208. doi: 10.1016/0168-8510(90)90421-9

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Huijg JM, Gebhardt WA, Crone MR, Dusseldorp E, Presseau J. Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research. Implementation Sci (2014) 9(11). doi: 10.1186/1748-5908-9-11

CrossRef Full Text | Google Scholar

38. Witmer BG, Singer MJ. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence-Teleop Virt (1998) 7(3):225–40. doi: 10.1162/105474698565686

CrossRef Full Text | Google Scholar

39. Overall J, Gorham D. The Brief Psychiatric Rating Scale. Psychol Rep (1962) 10:799–812. doi: 10.1159/000395069

CrossRef Full Text | Google Scholar

40. Nelson HE. National Adult Reading Test (NART): Test Manual. Windsor, UK: Nelson (1982).

Google Scholar

41. Arnett J, Labovitz J. Effect of physical layout in performance of the Trail Making Test. Psychol Assess (1995) 7(2):220–1. doi: 10.1037/1040-3590.7.2.220

CrossRef Full Text | Google Scholar

42. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol (2006) 3(2):77–101. doi: 10.1191/1478088706qp063oa

CrossRef Full Text | Google Scholar

43. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol (2013) 26(4):863. doi: 10.3389/fpsyg.2013.00863

CrossRef Full Text | Google Scholar

44. Hedges LV. Distribution Theory for Glass's Estimator of Effect Size and Related Estimators. J Educ Stat (1981) 6(2):107–28. doi: 10.2307/1164588

CrossRef Full Text | Google Scholar

45. Chow MCM. Factors influencing presence in virtual worlds. NI 2012 (2012) 2012:83. doi: 10.1089/cpb.2004.7.582

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Rus-Calafell M, Schneider S. Are we there yet?! A literature review of recent digital technology advances for the treatment of early psychosis. mHealth (2020) 6(3). doi: 10.21037/mhealth.2019.09.14

CrossRef Full Text | Google Scholar

47. Schulteis MT, Rothbaum BO. Ethical issues for the use of virtual reality in the psychological sciences. In: Bush S, Drexler M, editor. Ethical issues in clinical neuropsychology. Lisse, NL: Swets & Zeitlinger (2002). p. 243–80.

Google Scholar

48. Patel R, Oduola S, Callard F, Wykes T, Broadbent M, Stewart R, et al. What proportion of patients with psychosis is willing to take part in research? A mental health electronic case register analysis. BMJ Open (2017) 7:e013113. doi: 10.1136/bmjopen-2016-013113

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Wood L, Byrne R, Enache G, Morrison AP. A brief cognitive therapy intervention for internalised stigma in acute inpatients who experience psychosis: a feasibility randomised controlled trial. Psychiatry Res (2018) 262:303–10. doi: 10.1016/j.psychres.2017.12.030

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Kanniston KA, Korhonen J, Adams CE, Koivunen MH, Valhberg T, Valimaki MA. Factors associated with dropout during recruitment and follow-up periods of mHealth-based randomized controlled trial for Mobile.Net to encourage treatment adherence for people with serious mental health problems. J Med Internet Res (2017) 19(2):e46. doi: 10.2196/jmir.6417

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Schrank B, Brownell T, Jakaite Z, Larkin C, Pesola F, Riches S, et al. Evaluation of a positive psychotherapy group intervention for people with psychosis: pilot randomised controlled trial. Epidemiol Psychiatr Sci (2016) 25:235–46. doi: 10.1017/S2045796015000141

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Greer B, Robotham D, Simblett S, Curtis H, Griffiths H, Wykes T. Digital Exclusion Among Mental Health Service Users: Qualitative Investigation. J Med Internet Res (2019) 21(1):e11696. doi: 10.2196/11696

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Fletcher B, Gheorghe A, Moore D, Wilson S, Damery S. Improving the recruitment activity of clinicians in randomised controlled trials: a systematic review. BMJ Open (2012) 2(1):e000496. doi: 10.1136/bmjopen-2011-000496

PubMed Abstract | CrossRef Full Text | Google Scholar